Drop support for legacy ratelimit.proto and upgrade to v3 rls.proto #153
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Petr Pchelko <[email protected]>
erdody
pushed a commit
to medallia/ratelimit
that referenced
this pull request
Sep 24, 2020
…nvoyproxy#153) Signed-off-by: Petr Pchelko <[email protected]> Signed-off-by: Diego Erdody <[email protected]>
mjallday
added a commit
to verygoodsecurity/ratelimit
that referenced
this pull request
Jan 26, 2021
* Add Docker Compose File (#27) * docs: match envoy docs for remote_address ratelimiting (#29) * docs: document dependency on gostats (#30) * test and document whitelist behavior (#31) Signed-off-by: Daniel Hochman <[email protected]> * update dependencies (#35) * update dependencies * proto: use the proto defined in data-plane-api (#39) * proto: check in protos to allow importing ratelimit as a library (#40) * docs: update contact info (#42) * redis: add the option to use a separate redis pool for per second limits (#41) * fix duplicate mv (#43) * docker: upgrade docker-compose setup (#46) * Add gRPC health check (#47) * logging: set log level (#50) * go version: update to 1.11 (#53) * docker compose: expose gRPC port on docker compose setup (#55) * Configuration to ignore dotfiles. (#52) This allows ratelimit to run on Kubernetes with configuration from a configmap. * Add Dockerfile to enable builds (#58) Signed-off-by: Steve Sloka <[email protected]> * ci: fix build (#73) Fixes envoyproxy#71 Signed-off-by: Matt Klein <[email protected]> * Run unit and integration tests with race detector enabled (#65) * deps: update several of ratelimit's dependencies (#76) * add stalebot (#78) Signed-off-by: Matt Klein <[email protected]> * docs: fix example 4 sample config (#79) * fix redis-server binary name (envoyproxy#88) * Fix build Dockerfile (envoyproxy#98) Fix problem: src/service_cmd/runner/runner.go:10:2: cannot find package "github.com/lyft/ratelimit/proto/ratelimit" in any of: /usr/local/go/src/github.com/lyft/ratelimit/vendor/github.com/lyft/ratelimit/proto/ratelimit (vendor tree) /usr/local/go/src/vendor/github.com/lyft/ratelimit/proto/ratelimit /usr/local/go/src/github.com/lyft/ratelimit/proto/ratelimit (from $GOROOT) /go/src/github.com/lyft/ratelimit/proto/ratelimit (from $GOPATH) The command '/bin/sh -c go build -o /usr/local/bin/ratelimit src/service_cmd/main.go' returned a non-zero code: 1 * Redis TLS and Auth support (envoyproxy#96) This adds support for TLS connections to redis as well as support for authentication. Somewhat related to issue #61 * healthcheck: allow customizable healthcheck name (envoyproxy#102) Description: this patch allows a consumer of the server package to customize the name of the healthchecker. Signed-off-by: Jose Nino <[email protected]> * health: make a few more types public (envoyproxy#104) Description: envoyproxy#102 allowed for some customization. This PR makes the types public so that other servers can use this implementation. Signed-off-by: Jose Nino <[email protected]> * Add local cache to store whether it is over the limit (envoyproxy#111) * Plugin statstore into runner (envoyproxy#115) * fix: support auth without tls (envoyproxy#116) Signed-off-by: tangxinfa <[email protected]> * add local cache stats (envoyproxy#114) * Move license to templated Apache-2.0 (envoyproxy#123) Signed-off-by: Derek Schaller <[email protected]> * Enable go modules (envoyproxy#124) Signed-off-by: Steve Sloka <[email protected]> * CI: Github Actions (envoyproxy#127) Signed-off-by: Steve Sloka <[email protected]> * community: update contributing guide (envoyproxy#139) Fixes envoyproxy#138 Signed-off-by: Matt Klein <[email protected]> * add http 1 `/json` endpoint (envoyproxy#136) Signed-off-by: David Black <[email protected]> * Use mockgen version from go.mod instead of from "make bootstrap" (envoyproxy#143) Even though the Makefile wants to encourage using [email protected], it seems like the mocks have been generated using a pre-1.0 version of mockgen. Using "go run github.com/golang/mock/mockgen" as a go:generate command instead of just "mockgen" avoids the need to pre-install into the developer's $PATH and uses the go.mod-specified version Signed-off-by: David Weitzman <[email protected]> * Upgrade gostats dependency from 0.2.6 to 0.4.0 (envoyproxy#141) My interest is the UDP protocol support which appeared in gotstats 0.3.10 There's a breaking change as of https://github.com/lyft/gostats/releases/tag/v0.3.0 which is that gostats no longer publishes stats as expvars. Signed-off-by: David Weitzman <[email protected]> * Upgrade radix (envoyproxy#137) Signed-off-by: Tong Cai <[email protected]> * cache_impl_test.go: fix failing test with ipv6 (envoyproxy#144) A newly-added test in envoyproxy#137 checks the exact text of an error message which seems to vary when the network is tcp4 vs tcp6. This change relaxes the assertion to look for "connection refused" in a panic without making assumptions about what an IP address looks like. Example failure: --- FAIL: TestNewClientImpl (0.00s) --- FAIL: TestNewClientImpl/connection_refused (0.00s) cache_impl_test.go:442: Error Trace: cache_impl_test.go:442 Error: func (assert.PanicTestFunc)(0x1724110) should panic with error message: "dial tcp 127.0.0.1:12345: connect: connection refused" Panic value: "dial tcp [::1]:12345: connect: connection refused" Panic stack: goroutine 27 [running]: The testify assert package doesn't seem to support inexact matching on error messages, so the code gets a bit uglier than before. Signed-off-by: David Weitzman <[email protected]> * Split redis-specific logic from generic key-value store logic (envoyproxy#142) This is a pure refactoring with no behavior changes. It's a step toward being able to add memcache as a backend (see envoyproxy#140). This PR moves RateLimitCache from the redis package to a new "limiter" package, along with code for time/jitter, local cache stats, and constructing cache keys. All that can be reused with memcache. After this PR, the redis package is imported in exactly two places: - in service_cmd/runner/runner.go to call redis.NewRateLimiterCacheImplFromSettings() - in service/ratelimit.go in ShouldRateLimit to identify if a recovered panic is a redis.RedisError. If so, a stat is incremented and the panic() propagation is ended and in favor of returning the error as a the function result. The PR also includes changes by goimports to test/service/ratelimit_test.go so that the difference between package name vs file path name is explicit instead of implicit. Signed-off-by: David Weitzman <[email protected]> * json handler: return full ratelimit service response as json (envoyproxy#148) Previously an HTTP POST to /json would only return an HTTP status code, not all the other details supported by grpc ratelimit responses. With this change an HTTP POST to /json receives the full proto3 response encoded as json by jsonpb. It seems unlikely that anyone would be parsing the text "over limit" from the HTTP body instead of just reading the 429 response code, but for anyone doing that this would be a breaking change. Signed-off-by: David Weitzman <[email protected]> * Update goruntime to latest, 0.2.5. Add new config for watching changes in runtime config folder directly instead of the runtime root dir. (envoyproxy#151) Signed-off-by: Yuki Sawa <[email protected]> * Drop support for legacy ratelimit.proto and upgrade to v3 rls.proto (envoyproxy#153) Signed-off-by: Petr Pchelko <[email protected]> * Followups to v3 upgrade (envoyproxy#155) - Regenerate mocks based on new default protocol - Manually transform v2 messages to v3 messages - some of the fields were renamed thus json Marshal/Unmarshal does not work anymore - Added tests that verify conversion v2<->v3 works for headers fields - Update tests to use proto.Equal - simple assert.Equals might not work correctly for protobuf messages. Signed-off-by: Petr Pchelko <[email protected]> * Introduce a Dockerfile for running integration tests (envoyproxy#156) This diff creates Dockerfile.integration for running integration tests with clearly-defined dependencies. Previously the dependencies of the integration tests were defined within the github actions config. The new "make docker_tests" target should work for any developer with Docker installed. Previously there was no single command that would run integration tests across platforms, which makes development and onboarding harder. Even copying the command from github actions wouldn't have worked before, since that command quietly assumed that redis was already running on port 6379. Signed-off-by: David Weitzman <[email protected]> * Add support for rate limit overrides. (envoyproxy#158) Fixes envoyproxy#154 Signed-off-by: Petr Pchelko <[email protected]> * redis client: default to use explicit pipelining (envoyproxy#163) Signed-off-by: Tong Cai <[email protected]> * Clean go.mod file and update logrus to latest (envoyproxy#166) Signed-off-by: Yuki Sawa <[email protected]> * Add full test environment example. Fix bug in existing docker-compose. (envoyproxy#170) Signed-off-by: Yuki Sawa <[email protected]> * Implement LOG_FORMAT=json (envoyproxy#173) Centralized log collection system works better with logs in json format. E.g. DataDog strongly encourage setting up your logging library to produce your logs in JSON format to avoid the need for custom parsing rules. So, the next small fix is all we need to get json logs. Signed-off-by: Sergey Belyaev <[email protected]> * ci: Update github action to push docker image tagged with sha for each merge to master branch (envoyproxy#176) Updates the github action to also push a tagged image based upon the git sha. The tag also includes the current version of the release. Example tag: envoyproxy/ratelimit:f1758150b6dfed3e5c0ae13fb7bb6b8f6ae00b0e Fixes envoyproxy#174 Signed-off-by: Steve Sloka <[email protected]> * Update README.md to refer to existing files (envoyproxy#178) Signed-off-by: Margaret Gorguissian <[email protected]> * Add redis cluster and sentinel support (envoyproxy#179) Signed-off-by: Diego Erdody <[email protected]> * Add support for x-ratelimit-reset header (envoyproxy#182) Signed-off-by: Clara Andrew-Wani <[email protected]> * Create repokitteh.star (envoyproxy#187) Signed-off-by: Itay Donanhirsh <[email protected]> * refactor NearLimitRatio to environment variable (envoyproxy#186) Signed-off-by: zufardhiyaulhaq <[email protected]> * Fix flakey tests with DurationUntilReset. Update docker example to V3 config. (envoyproxy#192) Signed-off-by: Yuki Sawa <[email protected]> * Separate Redis cache and driver implementation (envoyproxy#194) Signed-off-by: William Albertus Dembo <[email protected]> * Set ratelimit filter to v3 api (envoyproxy#196) Signed-off-by: Yuki Sawa <[email protected]> * Add debug logging to indicate descriptor and limit (envoyproxy#197) Signed-off-by: Sasha Kulbii <[email protected]> * Implement BACKEND_TYPE=memcache as an alternative k/v store to redis (envoyproxy#172) MEMCACHE_HOST_PORT=host:port must be set with BACKEND_TYPE=memcache To minimize roundtrips when getting multiple keys, the memcache implementation does a GetMulti to fetch the existing rate limit usage and does increments asynchronously in background goroutines, since the memcache API doesn't offer multi-increment. Resolves envoyproxy#140 Signed-off-by: David Weitzman <[email protected]> * Refactoring of duplicated code across backend types (envoyproxy#202) Signed-off-by: Kateryna Nezdolii <[email protected]> * Small typo fix in README (envoyproxy#204) Signed-off-by: cpaika <[email protected]> * Circle CI (#1) * Add Circle Config * Touch to build * Move to expected path * Use Docker Command * Do it all here then * Env * Add Readme * Actual README * Add Docker Compose (#2) * VAULT-893 Fix docker-compose.yml * VAULT-893 Fix docker-compose.yml Co-authored-by: Marshall Jones <[email protected]> Co-authored-by: Jose Ulises Nino Rivera <[email protected]> Co-authored-by: Daniel Hochman <[email protected]> Co-authored-by: Martien Verbruggen <[email protected]> Co-authored-by: Ben Pope <[email protected]> Co-authored-by: Steve Sloka <[email protected]> Co-authored-by: Matt Klein <[email protected]> Co-authored-by: Charlie Vieth <[email protected]> Co-authored-by: Adil Hafeez <[email protected]> Co-authored-by: Kartograf <[email protected]> Co-authored-by: repl-david-winiarski <[email protected]> Co-authored-by: Junchao Lyu <[email protected]> Co-authored-by: tangxinfa <[email protected]> Co-authored-by: Steve Sloka <[email protected]> Co-authored-by: Matt Klein <[email protected]> Co-authored-by: dblackdblack <[email protected]> Co-authored-by: David Weitzman <[email protected]> Co-authored-by: Tong Cai <[email protected]> Co-authored-by: Yuki Sawa <[email protected]> Co-authored-by: Petr Pchelko <[email protected]> Co-authored-by: Petr Pchelko <[email protected]> Co-authored-by: Sergey Belyaev <[email protected]> Co-authored-by: Margaret G <[email protected]> Co-authored-by: Diego Erdody <[email protected]> Co-authored-by: Clara <[email protected]> Co-authored-by: Itay Donanhirsh <[email protected]> Co-authored-by: Zufar Dhiyaulhaq <[email protected]> Co-authored-by: William Albertus Dembo <[email protected]> Co-authored-by: Alex Kulbii <[email protected]> Co-authored-by: Kateryna Nezdolii <[email protected]> Co-authored-by: Christopher <[email protected]>
zdmytriv
pushed a commit
to verygoodsecurity/ratelimit
that referenced
this pull request
Jul 23, 2021
* Add Docker Compose File (#27) * docs: match envoy docs for remote_address ratelimiting (#29) * docs: document dependency on gostats (#30) * test and document whitelist behavior (#31) Signed-off-by: Daniel Hochman <[email protected]> * update dependencies (#35) * update dependencies * proto: use the proto defined in data-plane-api (#39) * proto: check in protos to allow importing ratelimit as a library (#40) * docs: update contact info (#42) * redis: add the option to use a separate redis pool for per second limits (#41) * fix duplicate mv (#43) * docker: upgrade docker-compose setup (#46) * Add gRPC health check (#47) * logging: set log level (#50) * go version: update to 1.11 (#53) * docker compose: expose gRPC port on docker compose setup (#55) * Configuration to ignore dotfiles. (#52) This allows ratelimit to run on Kubernetes with configuration from a configmap. * Add Dockerfile to enable builds (#58) Signed-off-by: Steve Sloka <[email protected]> * ci: fix build (#73) Fixes envoyproxy#71 Signed-off-by: Matt Klein <[email protected]> * Run unit and integration tests with race detector enabled (#65) * deps: update several of ratelimit's dependencies (#76) * add stalebot (#78) Signed-off-by: Matt Klein <[email protected]> * docs: fix example 4 sample config (#79) * fix redis-server binary name (envoyproxy#88) * Fix build Dockerfile (envoyproxy#98) Fix problem: src/service_cmd/runner/runner.go:10:2: cannot find package "github.com/lyft/ratelimit/proto/ratelimit" in any of: /usr/local/go/src/github.com/lyft/ratelimit/vendor/github.com/lyft/ratelimit/proto/ratelimit (vendor tree) /usr/local/go/src/vendor/github.com/lyft/ratelimit/proto/ratelimit /usr/local/go/src/github.com/lyft/ratelimit/proto/ratelimit (from $GOROOT) /go/src/github.com/lyft/ratelimit/proto/ratelimit (from $GOPATH) The command '/bin/sh -c go build -o /usr/local/bin/ratelimit src/service_cmd/main.go' returned a non-zero code: 1 * Redis TLS and Auth support (envoyproxy#96) This adds support for TLS connections to redis as well as support for authentication. Somewhat related to issue #61 * healthcheck: allow customizable healthcheck name (envoyproxy#102) Description: this patch allows a consumer of the server package to customize the name of the healthchecker. Signed-off-by: Jose Nino <[email protected]> * health: make a few more types public (envoyproxy#104) Description: envoyproxy#102 allowed for some customization. This PR makes the types public so that other servers can use this implementation. Signed-off-by: Jose Nino <[email protected]> * Add local cache to store whether it is over the limit (envoyproxy#111) * Plugin statstore into runner (envoyproxy#115) * fix: support auth without tls (envoyproxy#116) Signed-off-by: tangxinfa <[email protected]> * add local cache stats (envoyproxy#114) * Move license to templated Apache-2.0 (envoyproxy#123) Signed-off-by: Derek Schaller <[email protected]> * Enable go modules (envoyproxy#124) Signed-off-by: Steve Sloka <[email protected]> * CI: Github Actions (envoyproxy#127) Signed-off-by: Steve Sloka <[email protected]> * community: update contributing guide (envoyproxy#139) Fixes envoyproxy#138 Signed-off-by: Matt Klein <[email protected]> * add http 1 `/json` endpoint (envoyproxy#136) Signed-off-by: David Black <[email protected]> * Use mockgen version from go.mod instead of from "make bootstrap" (envoyproxy#143) Even though the Makefile wants to encourage using [email protected], it seems like the mocks have been generated using a pre-1.0 version of mockgen. Using "go run github.com/golang/mock/mockgen" as a go:generate command instead of just "mockgen" avoids the need to pre-install into the developer's $PATH and uses the go.mod-specified version Signed-off-by: David Weitzman <[email protected]> * Upgrade gostats dependency from 0.2.6 to 0.4.0 (envoyproxy#141) My interest is the UDP protocol support which appeared in gotstats 0.3.10 There's a breaking change as of https://github.com/lyft/gostats/releases/tag/v0.3.0 which is that gostats no longer publishes stats as expvars. Signed-off-by: David Weitzman <[email protected]> * Upgrade radix (envoyproxy#137) Signed-off-by: Tong Cai <[email protected]> * cache_impl_test.go: fix failing test with ipv6 (envoyproxy#144) A newly-added test in envoyproxy#137 checks the exact text of an error message which seems to vary when the network is tcp4 vs tcp6. This change relaxes the assertion to look for "connection refused" in a panic without making assumptions about what an IP address looks like. Example failure: --- FAIL: TestNewClientImpl (0.00s) --- FAIL: TestNewClientImpl/connection_refused (0.00s) cache_impl_test.go:442: Error Trace: cache_impl_test.go:442 Error: func (assert.PanicTestFunc)(0x1724110) should panic with error message: "dial tcp 127.0.0.1:12345: connect: connection refused" Panic value: "dial tcp [::1]:12345: connect: connection refused" Panic stack: goroutine 27 [running]: The testify assert package doesn't seem to support inexact matching on error messages, so the code gets a bit uglier than before. Signed-off-by: David Weitzman <[email protected]> * Split redis-specific logic from generic key-value store logic (envoyproxy#142) This is a pure refactoring with no behavior changes. It's a step toward being able to add memcache as a backend (see envoyproxy#140). This PR moves RateLimitCache from the redis package to a new "limiter" package, along with code for time/jitter, local cache stats, and constructing cache keys. All that can be reused with memcache. After this PR, the redis package is imported in exactly two places: - in service_cmd/runner/runner.go to call redis.NewRateLimiterCacheImplFromSettings() - in service/ratelimit.go in ShouldRateLimit to identify if a recovered panic is a redis.RedisError. If so, a stat is incremented and the panic() propagation is ended and in favor of returning the error as a the function result. The PR also includes changes by goimports to test/service/ratelimit_test.go so that the difference between package name vs file path name is explicit instead of implicit. Signed-off-by: David Weitzman <[email protected]> * json handler: return full ratelimit service response as json (envoyproxy#148) Previously an HTTP POST to /json would only return an HTTP status code, not all the other details supported by grpc ratelimit responses. With this change an HTTP POST to /json receives the full proto3 response encoded as json by jsonpb. It seems unlikely that anyone would be parsing the text "over limit" from the HTTP body instead of just reading the 429 response code, but for anyone doing that this would be a breaking change. Signed-off-by: David Weitzman <[email protected]> * Update goruntime to latest, 0.2.5. Add new config for watching changes in runtime config folder directly instead of the runtime root dir. (envoyproxy#151) Signed-off-by: Yuki Sawa <[email protected]> * Drop support for legacy ratelimit.proto and upgrade to v3 rls.proto (envoyproxy#153) Signed-off-by: Petr Pchelko <[email protected]> * Followups to v3 upgrade (envoyproxy#155) - Regenerate mocks based on new default protocol - Manually transform v2 messages to v3 messages - some of the fields were renamed thus json Marshal/Unmarshal does not work anymore - Added tests that verify conversion v2<->v3 works for headers fields - Update tests to use proto.Equal - simple assert.Equals might not work correctly for protobuf messages. Signed-off-by: Petr Pchelko <[email protected]> * Introduce a Dockerfile for running integration tests (envoyproxy#156) This diff creates Dockerfile.integration for running integration tests with clearly-defined dependencies. Previously the dependencies of the integration tests were defined within the github actions config. The new "make docker_tests" target should work for any developer with Docker installed. Previously there was no single command that would run integration tests across platforms, which makes development and onboarding harder. Even copying the command from github actions wouldn't have worked before, since that command quietly assumed that redis was already running on port 6379. Signed-off-by: David Weitzman <[email protected]> * Add support for rate limit overrides. (envoyproxy#158) Fixes envoyproxy#154 Signed-off-by: Petr Pchelko <[email protected]> * redis client: default to use explicit pipelining (envoyproxy#163) Signed-off-by: Tong Cai <[email protected]> * Clean go.mod file and update logrus to latest (envoyproxy#166) Signed-off-by: Yuki Sawa <[email protected]> * Add full test environment example. Fix bug in existing docker-compose. (envoyproxy#170) Signed-off-by: Yuki Sawa <[email protected]> * Implement LOG_FORMAT=json (envoyproxy#173) Centralized log collection system works better with logs in json format. E.g. DataDog strongly encourage setting up your logging library to produce your logs in JSON format to avoid the need for custom parsing rules. So, the next small fix is all we need to get json logs. Signed-off-by: Sergey Belyaev <[email protected]> * ci: Update github action to push docker image tagged with sha for each merge to master branch (envoyproxy#176) Updates the github action to also push a tagged image based upon the git sha. The tag also includes the current version of the release. Example tag: envoyproxy/ratelimit:f1758150b6dfed3e5c0ae13fb7bb6b8f6ae00b0e Fixes envoyproxy#174 Signed-off-by: Steve Sloka <[email protected]> * Update README.md to refer to existing files (envoyproxy#178) Signed-off-by: Margaret Gorguissian <[email protected]> * Add redis cluster and sentinel support (envoyproxy#179) Signed-off-by: Diego Erdody <[email protected]> * Add support for x-ratelimit-reset header (envoyproxy#182) Signed-off-by: Clara Andrew-Wani <[email protected]> * Create repokitteh.star (envoyproxy#187) Signed-off-by: Itay Donanhirsh <[email protected]> * refactor NearLimitRatio to environment variable (envoyproxy#186) Signed-off-by: zufardhiyaulhaq <[email protected]> * Fix flakey tests with DurationUntilReset. Update docker example to V3 config. (envoyproxy#192) Signed-off-by: Yuki Sawa <[email protected]> * Separate Redis cache and driver implementation (envoyproxy#194) Signed-off-by: William Albertus Dembo <[email protected]> * Set ratelimit filter to v3 api (envoyproxy#196) Signed-off-by: Yuki Sawa <[email protected]> * Add debug logging to indicate descriptor and limit (envoyproxy#197) Signed-off-by: Sasha Kulbii <[email protected]> * Implement BACKEND_TYPE=memcache as an alternative k/v store to redis (envoyproxy#172) MEMCACHE_HOST_PORT=host:port must be set with BACKEND_TYPE=memcache To minimize roundtrips when getting multiple keys, the memcache implementation does a GetMulti to fetch the existing rate limit usage and does increments asynchronously in background goroutines, since the memcache API doesn't offer multi-increment. Resolves envoyproxy#140 Signed-off-by: David Weitzman <[email protected]> * Refactoring of duplicated code across backend types (envoyproxy#202) Signed-off-by: Kateryna Nezdolii <[email protected]> * Small typo fix in README (envoyproxy#204) Signed-off-by: cpaika <[email protected]> * Circle CI (#1) * Add Circle Config * Touch to build * Move to expected path * Use Docker Command * Do it all here then * Env * Add Readme * Actual README * Add Docker Compose (#2) * VAULT-893 Fix docker-compose.yml * VAULT-893 Fix docker-compose.yml Co-authored-by: Marshall Jones <[email protected]> Co-authored-by: Jose Ulises Nino Rivera <[email protected]> Co-authored-by: Daniel Hochman <[email protected]> Co-authored-by: Martien Verbruggen <[email protected]> Co-authored-by: Ben Pope <[email protected]> Co-authored-by: Steve Sloka <[email protected]> Co-authored-by: Matt Klein <[email protected]> Co-authored-by: Charlie Vieth <[email protected]> Co-authored-by: Adil Hafeez <[email protected]> Co-authored-by: Kartograf <[email protected]> Co-authored-by: repl-david-winiarski <[email protected]> Co-authored-by: Junchao Lyu <[email protected]> Co-authored-by: tangxinfa <[email protected]> Co-authored-by: Steve Sloka <[email protected]> Co-authored-by: Matt Klein <[email protected]> Co-authored-by: dblackdblack <[email protected]> Co-authored-by: David Weitzman <[email protected]> Co-authored-by: Tong Cai <[email protected]> Co-authored-by: Yuki Sawa <[email protected]> Co-authored-by: Petr Pchelko <[email protected]> Co-authored-by: Petr Pchelko <[email protected]> Co-authored-by: Sergey Belyaev <[email protected]> Co-authored-by: Margaret G <[email protected]> Co-authored-by: Diego Erdody <[email protected]> Co-authored-by: Clara <[email protected]> Co-authored-by: Itay Donanhirsh <[email protected]> Co-authored-by: Zufar Dhiyaulhaq <[email protected]> Co-authored-by: William Albertus Dembo <[email protected]> Co-authored-by: Alex Kulbii <[email protected]> Co-authored-by: Kateryna Nezdolii <[email protected]> Co-authored-by: Christopher <[email protected]>
zdmytriv
pushed a commit
to verygoodsecurity/ratelimit
that referenced
this pull request
Aug 2, 2021
* Add Docker Compose File (#27) * docs: match envoy docs for remote_address ratelimiting (#29) * docs: document dependency on gostats (#30) * test and document whitelist behavior (#31) Signed-off-by: Daniel Hochman <[email protected]> * update dependencies (#35) * update dependencies * proto: use the proto defined in data-plane-api (#39) * proto: check in protos to allow importing ratelimit as a library (#40) * docs: update contact info (#42) * redis: add the option to use a separate redis pool for per second limits (#41) * fix duplicate mv (#43) * docker: upgrade docker-compose setup (#46) * Add gRPC health check (#47) * logging: set log level (#50) * go version: update to 1.11 (#53) * docker compose: expose gRPC port on docker compose setup (#55) * Configuration to ignore dotfiles. (#52) This allows ratelimit to run on Kubernetes with configuration from a configmap. * Add Dockerfile to enable builds (#58) Signed-off-by: Steve Sloka <[email protected]> * ci: fix build (#73) Fixes envoyproxy#71 Signed-off-by: Matt Klein <[email protected]> * Run unit and integration tests with race detector enabled (#65) * deps: update several of ratelimit's dependencies (#76) * add stalebot (#78) Signed-off-by: Matt Klein <[email protected]> * docs: fix example 4 sample config (#79) * fix redis-server binary name (envoyproxy#88) * Fix build Dockerfile (envoyproxy#98) Fix problem: src/service_cmd/runner/runner.go:10:2: cannot find package "github.com/lyft/ratelimit/proto/ratelimit" in any of: /usr/local/go/src/github.com/lyft/ratelimit/vendor/github.com/lyft/ratelimit/proto/ratelimit (vendor tree) /usr/local/go/src/vendor/github.com/lyft/ratelimit/proto/ratelimit /usr/local/go/src/github.com/lyft/ratelimit/proto/ratelimit (from $GOROOT) /go/src/github.com/lyft/ratelimit/proto/ratelimit (from $GOPATH) The command '/bin/sh -c go build -o /usr/local/bin/ratelimit src/service_cmd/main.go' returned a non-zero code: 1 * Redis TLS and Auth support (envoyproxy#96) This adds support for TLS connections to redis as well as support for authentication. Somewhat related to issue #61 * healthcheck: allow customizable healthcheck name (envoyproxy#102) Description: this patch allows a consumer of the server package to customize the name of the healthchecker. Signed-off-by: Jose Nino <[email protected]> * health: make a few more types public (envoyproxy#104) Description: envoyproxy#102 allowed for some customization. This PR makes the types public so that other servers can use this implementation. Signed-off-by: Jose Nino <[email protected]> * Add local cache to store whether it is over the limit (envoyproxy#111) * Plugin statstore into runner (envoyproxy#115) * fix: support auth without tls (envoyproxy#116) Signed-off-by: tangxinfa <[email protected]> * add local cache stats (envoyproxy#114) * Move license to templated Apache-2.0 (envoyproxy#123) Signed-off-by: Derek Schaller <[email protected]> * Enable go modules (envoyproxy#124) Signed-off-by: Steve Sloka <[email protected]> * CI: Github Actions (envoyproxy#127) Signed-off-by: Steve Sloka <[email protected]> * community: update contributing guide (envoyproxy#139) Fixes envoyproxy#138 Signed-off-by: Matt Klein <[email protected]> * add http 1 `/json` endpoint (envoyproxy#136) Signed-off-by: David Black <[email protected]> * Use mockgen version from go.mod instead of from "make bootstrap" (envoyproxy#143) Even though the Makefile wants to encourage using [email protected], it seems like the mocks have been generated using a pre-1.0 version of mockgen. Using "go run github.com/golang/mock/mockgen" as a go:generate command instead of just "mockgen" avoids the need to pre-install into the developer's $PATH and uses the go.mod-specified version Signed-off-by: David Weitzman <[email protected]> * Upgrade gostats dependency from 0.2.6 to 0.4.0 (envoyproxy#141) My interest is the UDP protocol support which appeared in gotstats 0.3.10 There's a breaking change as of https://github.com/lyft/gostats/releases/tag/v0.3.0 which is that gostats no longer publishes stats as expvars. Signed-off-by: David Weitzman <[email protected]> * Upgrade radix (envoyproxy#137) Signed-off-by: Tong Cai <[email protected]> * cache_impl_test.go: fix failing test with ipv6 (envoyproxy#144) A newly-added test in envoyproxy#137 checks the exact text of an error message which seems to vary when the network is tcp4 vs tcp6. This change relaxes the assertion to look for "connection refused" in a panic without making assumptions about what an IP address looks like. Example failure: --- FAIL: TestNewClientImpl (0.00s) --- FAIL: TestNewClientImpl/connection_refused (0.00s) cache_impl_test.go:442: Error Trace: cache_impl_test.go:442 Error: func (assert.PanicTestFunc)(0x1724110) should panic with error message: "dial tcp 127.0.0.1:12345: connect: connection refused" Panic value: "dial tcp [::1]:12345: connect: connection refused" Panic stack: goroutine 27 [running]: The testify assert package doesn't seem to support inexact matching on error messages, so the code gets a bit uglier than before. Signed-off-by: David Weitzman <[email protected]> * Split redis-specific logic from generic key-value store logic (envoyproxy#142) This is a pure refactoring with no behavior changes. It's a step toward being able to add memcache as a backend (see envoyproxy#140). This PR moves RateLimitCache from the redis package to a new "limiter" package, along with code for time/jitter, local cache stats, and constructing cache keys. All that can be reused with memcache. After this PR, the redis package is imported in exactly two places: - in service_cmd/runner/runner.go to call redis.NewRateLimiterCacheImplFromSettings() - in service/ratelimit.go in ShouldRateLimit to identify if a recovered panic is a redis.RedisError. If so, a stat is incremented and the panic() propagation is ended and in favor of returning the error as a the function result. The PR also includes changes by goimports to test/service/ratelimit_test.go so that the difference between package name vs file path name is explicit instead of implicit. Signed-off-by: David Weitzman <[email protected]> * json handler: return full ratelimit service response as json (envoyproxy#148) Previously an HTTP POST to /json would only return an HTTP status code, not all the other details supported by grpc ratelimit responses. With this change an HTTP POST to /json receives the full proto3 response encoded as json by jsonpb. It seems unlikely that anyone would be parsing the text "over limit" from the HTTP body instead of just reading the 429 response code, but for anyone doing that this would be a breaking change. Signed-off-by: David Weitzman <[email protected]> * Update goruntime to latest, 0.2.5. Add new config for watching changes in runtime config folder directly instead of the runtime root dir. (envoyproxy#151) Signed-off-by: Yuki Sawa <[email protected]> * Drop support for legacy ratelimit.proto and upgrade to v3 rls.proto (envoyproxy#153) Signed-off-by: Petr Pchelko <[email protected]> * Followups to v3 upgrade (envoyproxy#155) - Regenerate mocks based on new default protocol - Manually transform v2 messages to v3 messages - some of the fields were renamed thus json Marshal/Unmarshal does not work anymore - Added tests that verify conversion v2<->v3 works for headers fields - Update tests to use proto.Equal - simple assert.Equals might not work correctly for protobuf messages. Signed-off-by: Petr Pchelko <[email protected]> * Introduce a Dockerfile for running integration tests (envoyproxy#156) This diff creates Dockerfile.integration for running integration tests with clearly-defined dependencies. Previously the dependencies of the integration tests were defined within the github actions config. The new "make docker_tests" target should work for any developer with Docker installed. Previously there was no single command that would run integration tests across platforms, which makes development and onboarding harder. Even copying the command from github actions wouldn't have worked before, since that command quietly assumed that redis was already running on port 6379. Signed-off-by: David Weitzman <[email protected]> * Add support for rate limit overrides. (envoyproxy#158) Fixes envoyproxy#154 Signed-off-by: Petr Pchelko <[email protected]> * redis client: default to use explicit pipelining (envoyproxy#163) Signed-off-by: Tong Cai <[email protected]> * Clean go.mod file and update logrus to latest (envoyproxy#166) Signed-off-by: Yuki Sawa <[email protected]> * Add full test environment example. Fix bug in existing docker-compose. (envoyproxy#170) Signed-off-by: Yuki Sawa <[email protected]> * Implement LOG_FORMAT=json (envoyproxy#173) Centralized log collection system works better with logs in json format. E.g. DataDog strongly encourage setting up your logging library to produce your logs in JSON format to avoid the need for custom parsing rules. So, the next small fix is all we need to get json logs. Signed-off-by: Sergey Belyaev <[email protected]> * ci: Update github action to push docker image tagged with sha for each merge to master branch (envoyproxy#176) Updates the github action to also push a tagged image based upon the git sha. The tag also includes the current version of the release. Example tag: envoyproxy/ratelimit:f1758150b6dfed3e5c0ae13fb7bb6b8f6ae00b0e Fixes envoyproxy#174 Signed-off-by: Steve Sloka <[email protected]> * Update README.md to refer to existing files (envoyproxy#178) Signed-off-by: Margaret Gorguissian <[email protected]> * Add redis cluster and sentinel support (envoyproxy#179) Signed-off-by: Diego Erdody <[email protected]> * Add support for x-ratelimit-reset header (envoyproxy#182) Signed-off-by: Clara Andrew-Wani <[email protected]> * Create repokitteh.star (envoyproxy#187) Signed-off-by: Itay Donanhirsh <[email protected]> * refactor NearLimitRatio to environment variable (envoyproxy#186) Signed-off-by: zufardhiyaulhaq <[email protected]> * Fix flakey tests with DurationUntilReset. Update docker example to V3 config. (envoyproxy#192) Signed-off-by: Yuki Sawa <[email protected]> * Separate Redis cache and driver implementation (envoyproxy#194) Signed-off-by: William Albertus Dembo <[email protected]> * Set ratelimit filter to v3 api (envoyproxy#196) Signed-off-by: Yuki Sawa <[email protected]> * Add debug logging to indicate descriptor and limit (envoyproxy#197) Signed-off-by: Sasha Kulbii <[email protected]> * Implement BACKEND_TYPE=memcache as an alternative k/v store to redis (envoyproxy#172) MEMCACHE_HOST_PORT=host:port must be set with BACKEND_TYPE=memcache To minimize roundtrips when getting multiple keys, the memcache implementation does a GetMulti to fetch the existing rate limit usage and does increments asynchronously in background goroutines, since the memcache API doesn't offer multi-increment. Resolves envoyproxy#140 Signed-off-by: David Weitzman <[email protected]> * Refactoring of duplicated code across backend types (envoyproxy#202) Signed-off-by: Kateryna Nezdolii <[email protected]> * Small typo fix in README (envoyproxy#204) Signed-off-by: cpaika <[email protected]> * Circle CI (#1) * Add Circle Config * Touch to build * Move to expected path * Use Docker Command * Do it all here then * Env * Add Readme * Actual README * Add Docker Compose (#2) * VAULT-893 Fix docker-compose.yml * VAULT-893 Fix docker-compose.yml Co-authored-by: Marshall Jones <[email protected]> Co-authored-by: Jose Ulises Nino Rivera <[email protected]> Co-authored-by: Daniel Hochman <[email protected]> Co-authored-by: Martien Verbruggen <[email protected]> Co-authored-by: Ben Pope <[email protected]> Co-authored-by: Steve Sloka <[email protected]> Co-authored-by: Matt Klein <[email protected]> Co-authored-by: Charlie Vieth <[email protected]> Co-authored-by: Adil Hafeez <[email protected]> Co-authored-by: Kartograf <[email protected]> Co-authored-by: repl-david-winiarski <[email protected]> Co-authored-by: Junchao Lyu <[email protected]> Co-authored-by: tangxinfa <[email protected]> Co-authored-by: Steve Sloka <[email protected]> Co-authored-by: Matt Klein <[email protected]> Co-authored-by: dblackdblack <[email protected]> Co-authored-by: David Weitzman <[email protected]> Co-authored-by: Tong Cai <[email protected]> Co-authored-by: Yuki Sawa <[email protected]> Co-authored-by: Petr Pchelko <[email protected]> Co-authored-by: Petr Pchelko <[email protected]> Co-authored-by: Sergey Belyaev <[email protected]> Co-authored-by: Margaret G <[email protected]> Co-authored-by: Diego Erdody <[email protected]> Co-authored-by: Clara <[email protected]> Co-authored-by: Itay Donanhirsh <[email protected]> Co-authored-by: Zufar Dhiyaulhaq <[email protected]> Co-authored-by: William Albertus Dembo <[email protected]> Co-authored-by: Alex Kulbii <[email protected]> Co-authored-by: Kateryna Nezdolii <[email protected]> Co-authored-by: Christopher <[email protected]>
timcovar
pushed a commit
to goatapp/ratelimit
that referenced
this pull request
Jan 16, 2024
…nvoyproxy#153) Signed-off-by: Petr Pchelko <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #152.
Signed-off-by: Petr Pchelko [email protected]