This repository has been archived by the owner on Nov 2, 2023. It is now read-only.
v1.0.0 #173
Merged
v1.0.0 #173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
September 30, 2020 15:58
Fixes - (#158) PII: make the PII scrubbing of In-App WAF attack events case-insensitive in order to correctly scrub transformed request parameters. - (#159) Monitoring: fix the content type and length monitoring of HTTP responses. - (#157) Gin middleware: use the request Go context instead of Gin's so that the agent can properly manage the request execution context, but also to correctly propagate values stored in the Go context before the middleware function.
Add a new type of metrics for binning metrics and explicitely rename the previous sum metrics store types to make the API clearer.
Add a thread-safe shared stopwatch implementation accouting time duration between the first goroutine starting it and the last stopping. It will be used to compute sqreen's execution time per request for the simplest preformance monitoring level 1. It indeed allows to read time less frequenttl than multiple detailed timers.
Introduce a complete callback framework providing a fully managed and abstraced callback-implementation API. Two new interfaces were introduced for that: 1. The RuleContext is the interface with the security rule the callback is serving. It hides implementation details such as the rule name, the blocking mode, etc. This context is provided at callback instantiation. 2. The CallbackContext is the result of a RuleContext within a given ProtectionContext. It is obtained by calling the RuleContext's Pre() and Post() methods, both expecting a function closure having the CallbackContext as argument. And this is how the callback gets wrapped with rule- and protection-specific features. The closure also allows to do the bridge with the hooked function for example to set its return values when blocking. This patch ports every callback to this new architecture.
Some improvements - still not perfect - simplifying the protection API and expectations regarding the agent.
…ction context changes
added 24 commits
October 30, 2020 13:42
Monitor the return error of Echo request handlers to see if they are Echo's HTTPErrors, so that we can properly capture the HTTP status code that will be later used by Echo. The HTTP response is otherwise out of the middleware scope in that case.
Add backoff error logging for better error management in the request hotpaths. A backoff error logger is created using `plog.WithBackoff()`. It manages error counters and logs the error every time the counter value is a power of two. Error can be either individually counted when wrapped by `sqerrors.WithKey()`, or simply share the common counter of the backoff error logger object.
Simple backoff error logger integration in the agent and its callbacks. The error indexing should be further defined over time as it is currently oversampling due to shared keys (eg. avoiding using the same key for two distinct binding accessor expresssions).
Critical execution errors happening in the request hot path are now logged according to exponential backoff counters of their occurrences. This avoids slowing down request handlers but also spamming the internal agent queue. Such error will now go through the agent error logging facility every time the backoff counter is a power of two (1, 2, 4, 8, etc.). This is disabled when in debug log level.
Correctly handle the default response case.
Fix the response's content-length monitoring with the default response (ie. the handler does nothing).
… the tests Use the track event name the backend expects. Restoring the tests highlighted some inconsistent blocking behavior between user/ip blocking/redirecting that are now fixed.
…y response actions Enable the correct event display on the dashboard by fixing the track event name with the one the backend actually expects. Also, restoring the security response tests highlighted some inconsistent blocking behavior between user/ip blocking/redirecting that are now fixed.
Moving to an interface-only public API has the benefit of hiding the type-implementation details that may change over time in the future. The returned SDK values are now interface values and no longer pointer values. This may break integrations having explicit types. We recommend to instead use type-inference when possible. From now, this change will allow us to transparently change the underlying values returned by SDK calls. For example, as of today, the underlying SDK values are small event collection wrappers that shouldn't be returned by reference in order to avoid useless allocations, for the benefit of the overall application performance. If the underlying object size increase in the future, we will be able to return allocated structures instead without breaking existing SDK integrations.
Moving to an interface-only public API has the benefit of hiding the type-implementation details that may change over time in the future. The returned SDK values are now interface values and no longer pointer values. This may break integrations having explicit types. We recommend to instead use type-inference when possible. From now, this change will allow us to transparently change the underlying values returned by SDK calls. For example, as of today, the underlying SDK values are small event collection wrappers that shouldn't be returned by reference in order to avoid useless allocations, for the benefit of the overall application performance. If the underlying object size increase in the future, we will be able to return allocated structures instead without breaking existing SDK integrations.
…re reading its package path Following up on elastic/apm-agent-go#848 and the addition of the `Unwrap()` method to Elastic's SQL driver tracer, it is now possible to unwrap it and properly read the package path of the underlying driver as expected.
…method errors Add finer-grained error keys that can be returned by the SQL dialect getter method. This also highlighted the fact errors can be assigned multiple keys while bubbling up the call stack. Therefore, the quick solution to get an error key is to make `sqerrors.Key()` return the deepest key rather than the top one. Ideally, it should rather combine multiple keys together in the future.
…re reading its package path Following up on elastic/apm-agent-go#848 and the addition of the `Unwrap()` method to Elastic's SQL driver tracer, it is now possible to unwrap it and properly read the package path of the underlying driver as expected.
Make the HTTP response writer wrapper transparent by implementing the same *known* interfaces as the underlying HTTP response writer it implements. The list of interfaces is currently every optional `net/http` interfaces, and some from `io` when relevant: - `http.Flusher`: to allow flushing any buffered to the client. This enables support for streaming handlers. - `http.Hijacker`: to allow handlers to takeover the HTTP connection. This should enable the support for websocket servers, which are not officially supported by Sqreen, but is now experimentally allowed. - `http.Pusher`: for HTTP2 server push. - `http.CloseNotifier`: the deprecated closed connection notifier. - `io.ReaderFrom`: for optimized copies (eg. `io.Copy(file, w)`) - `io.WriteString`: for optimized string write (which avoids a temporary string copy into a byte slice) The transparent wrapper implementation has been generated from a tool that will be released in the Go agent repository in the future. Fixes #162 and #134
Remove the useless response writer wrapper as Echo and Gin already monitor them. Note that Gin actually doesn't allow changing the writter and all its helpers use the internal in-memory writer. Therefore, we are for now accepting multiple responses to keep the original response writer behaviour.
Make the HTTP response writer wrapper transparent by implementing the same
*known* interfaces as the underlying HTTP response writer. The
list of interfaces is currently every optional `net/http` interfaces, and some
from `io` when relevant:
- `http.Flusher`: to allow flushing any buffered to the client. This enables
support for streaming handlers.
- `http.Hijacker`: to allow handlers to takeover the HTTP connection. This
should enable the support for websocket servers, which are not officially
supported by Sqreen, but is now experimentally allowed.
- `http.Pusher`: for HTTP2 server push.
- `http.CloseNotifier`: the deprecated closed connection notifier.
- `io.ReaderFrom`: for optimized copies (eg. `io.Copy(file, w)`)
- `io.WriteString`: for optimized string write (which avoids a temporary string copy into a byte slice)
The transparent wrapper implementation has been generated with a tool that will
be released in the Go agent repository in the future.
Fixes #162 and #134
- Restore max length checks - Update max length error checks
Add `sdk.FromRequest()` helper function to retrieve Sqreen's SDK handle from a request. It is a shortcut to `sdk.FromContext(r.Context())`.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New Features
(sdk: add FromRequest() helper #172) New SDK convenience function:
Add a new helper function
sdk.FromRequest()allowing to retrieve Sqreen'srequest context and perform SDK calls. It is equivalent to
sdk.FromContext(r.Context()).(Performance monitoring #156) Performance monitoring:
Monitor the execution time of requests protected by Sqreen. Optionally, it is
possible to enforce the maximum amount of time Sqreen is allowed to run per
request: Sqreen's monitoring and protections will only run for the given
amount of time. This option is disabled by default and should be used with
caution as it can lead to partially protected requests.
The resulting performance monitoring diagrams and setting are available at
https://my.sqreen.com/application/goto/settings/performance.
Note that the execution time diagram cannot be used as a strict Application
Performance Monitoring diagram as it is based on a lossy representation. It
gives rough estimates of the actual execution time.
(sdk/middleware/sqhttp: transparent http response writer wrapper #170) Transparent response writer instrumentation:
Make the HTTP response writer instrumentation transparent by providing the
same set of interfaces as the instrumented HTTP response writer. The set of
interfaces is currently every optional
net/httpresponse writer interface,along with some relevant
iointerfaces, among which:http.Flusher: for HTTP streaming support (multipart, chunked...).http.Pusher: for HTTP2 server push support.http.Hijacker: for websocket server support (experimental).io.ReaderFrom: for optimized copies (eg. file copies)io.WriteString: for optimized string copies.(HTTP Status Code Monitoring: NotFound attack event #163) HTTP status code 404 (not found) monitoring:
Automatically log a security event when the response status code is 404. This
event is used by an internal Sqreen backend playbook to detect security scans.
(HTTP Status Code Monitoring: NotFound attack event #163) Scalable security event throughput:
To be able to handle a higher throughput of security events, the agent can now
scale its number of goroutines. An extra goroutine is created every time the
internal event queue is full, up to the number of available CPUs. Note that
the agent still drops security events when the event queue is full in order to
avoid slowing down the host application.
(Hot-path error management #165) Agent errors in the request hot-path:
To avoid slowing down request handlers, agent errors happening in the request
hot path are now logged based on an exponential backoff algorithm.
This is disabled when the agent log level is
debug.Breaking Change
The SDK function and method return values are no longer pointer values but Go
interface values. This may break integrations using explicit SDK return types,
and we recommend to instead use type-inference when possible. This change will
allow us to transparently change the actual return values without involving
any further breaking change.
As of today, the actual return value is a structure small enough to be
returned by value in order to save memory-allocation and garbage-collection
time. Returning an interface value allows to hide such implementation detail.
Fixes
(Security Responses: fix the name of security response action events #167) Playbook security response events:
Fix playbook security response events (blocking or redirecting a user or ip)
so that Sqreen's dashboard can properly display them and link them to their
source playbook.
(agent/rule/callback/binding-accessors/sql: unwrap the SQL driver before reading its package path #169) SQL-injection protection with Elastic APM:
Fix the detection of the SQL dialect when the SQL driver is instrumented by
Elastic's APM tracer. This requires Elastic's Go agent version greater than
v1.9.0.(sdk/middleware/echo: fix status code monitoring of handler errors #164) Echo middleware:
Fix the response status code monitoring when Echo's request handlers return an
error.
(sdk/middleware/sqgin: response's content length monitoring #166) Gin middleware:
Fix the response content-length monitoring of default responses
(ie. when the handler does nothing).