Skip to content
This repository has been archived by the owner on Nov 2, 2023. It is now read-only.

v1.0.0 #173

Merged
merged 67 commits into from
Nov 19, 2020
Merged

v1.0.0 #173

merged 67 commits into from
Nov 19, 2020

Commits on Sep 30, 2020

  1. v0.16.1

    Fixes
    
    - (#158) PII: make the PII scrubbing of In-App WAF attack events
      case-insensitive in order to correctly scrub transformed request parameters.
    
    - (#159) Monitoring: fix the content type and length monitoring of HTTP
      responses.
    
    - (#157) Gin middleware: use the request Go context instead of Gin's so that the
      agent can properly manage the request execution context, but also to correctly
      propagate values stored in the Go context before the middleware function.
    Julio Guerra committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    378102d View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. ci: add a codeql analysis

    Julio Guerra committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    e530ae8 View commit details
    Browse the repository at this point in the history
  2. ci: add github's codeql analysis

    Julio Guerra authored Oct 1, 2020
    Configuration menu
    Copy the full SHA
    9451a76 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2020

  1. agent/metrics: add binning metrics

    Add a new type of metrics for binning metrics and explicitely rename the
    previous sum metrics store types to make the API clearer.
    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    e688a38 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e74b9a0 View commit details
    Browse the repository at this point in the history
  3. agent: performance histogram integration

    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    bc3cd5d View commit details
    Browse the repository at this point in the history
  4. sqlib/sqtime: add a shared stopwatch

    Add a thread-safe shared stopwatch implementation accouting time duration
    between the first goroutine starting it and the last stopping. It will be used
    to compute sqreen's execution time per request for the simplest preformance
    monitoring level 1. It indeed allows to read time less frequenttl than multiple
    detailed timers.
    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    c57c89a View commit details
    Browse the repository at this point in the history
  5. agent/rule/callback/waf: refactor the callbacks to the new native cal…

    …lback api
    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    220fd2c View commit details
    Browse the repository at this point in the history
  6. agent/rule/callback: callback framework

    Introduce a complete callback framework providing a fully managed and abstraced
    callback-implementation API. Two new interfaces were introduced for that:
    
    1. The RuleContext is the interface with the security rule the callback is
       serving. It hides implementation details such as the rule name, the blocking
       mode, etc. This context is provided at callback instantiation.
    
    2. The CallbackContext is the result of a RuleContext within a given
       ProtectionContext. It is obtained by calling the RuleContext's Pre() and
       Post() methods, both expecting a function closure having the CallbackContext
       as argument. And this is how the callback gets wrapped with rule- and
       protection-specific features. The closure also allows to do the bridge with
       the hooked function for example to set its return values when blocking.
    
    This patch ports every callback to this new architecture.
    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    c11f415 View commit details
    Browse the repository at this point in the history
  7. agent: performance cap poc

    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    d71cf72 View commit details
    Browse the repository at this point in the history
  8. go: update go-sdk to v1.2.0 including binning metrics

    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    5934786 View commit details
    Browse the repository at this point in the history
  9. agent: refactoring protection/agent interfaces

    Some improvements - still not perfect - simplifying the protection API and
    expectations regarding the agent.
    Julio Guerra committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    a794b40 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2020

  1. agent/rule/callback: fix the callcount callback wrapper

    Julio Guerra committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    3e01632 View commit details
    Browse the repository at this point in the history
  2. sdk/middleware/sqecho/v4: update the middleware with the recent prote…

    …ction context changes
    Julio Guerra committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    2876050 View commit details
    Browse the repository at this point in the history
  3. agent/protection/http: fix tests

    Julio Guerra committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    15ca756 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2020

  1. sdk/middleware: fix broken tests

    Julio Guerra committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    7583241 View commit details
    Browse the repository at this point in the history
  2. agent/protection/http: fix http protection tests

    Julio Guerra committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    6f337da View commit details
    Browse the repository at this point in the history
  3. agent/command: fix tests

    Julio Guerra committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    229e82c View commit details
    Browse the repository at this point in the history
  4. agent/backend/api: disable flaky fuzzing test

    Julio Guerra committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    70c0426 View commit details
    Browse the repository at this point in the history
  5. agent/rule/callback: fix callback tests

    Julio Guerra committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    deb4bda View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2020

  1. sqlib/sqtime: fix first/last start/stop race

    Julio Guerra committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    d264bb0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f7414d1 View commit details
    Browse the repository at this point in the history
  3. agent/metrics: fix ongoing values handling

    Julio Guerra committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    1caf4d2 View commit details
    Browse the repository at this point in the history
  4. agent/metrics: update metrics documentation

    Julio Guerra committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    4d95c3a View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2020

  1. Configuration menu
    Copy the full SHA
    0869c28 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2020

  1. agent/rule/callback: add tests of callback middlewares

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    0df2594 View commit details
    Browse the repository at this point in the history
  2. agent/rule/callback: fix go1.12 compilation error

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    fedf3fa View commit details
    Browse the repository at this point in the history
  3. agent/rule/callback: try fix flaky windows test

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    6e0c42a View commit details
    Browse the repository at this point in the history
  4. agent/rule/callback: try fix flaky windows test

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    1072066 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    66d6f39 View commit details
    Browse the repository at this point in the history
  6. agent/metrics: try debug windows panic

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    8b3df8e View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    bdf2d9b View commit details
    Browse the repository at this point in the history
  8. agent/metrics: fix max value when ongoing

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    2b73910 View commit details
    Browse the repository at this point in the history
  9. agent/rule/waf: add some deadline checks to abort the waf when the de…

    …adline exceeded
    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    41c8af8 View commit details
    Browse the repository at this point in the history
  10. agent/metrics: remove debug block of code

    Julio Guerra committed Oct 22, 2020
    Configuration menu
    Copy the full SHA
    7c8604f View commit details
    Browse the repository at this point in the history
  11. agent: performance monitoring

    Monitor the execution time of requests protected by Sqreen. Optionally, it is
    now possible to set the maximum amount of time Sqreen is allowed to run per
    request: Sqreen's monitoring and protections will only run for the given amount
    of time. This option is disabled by default and should be used with caution as
    it can lead to partially protected requests.
    
    The resulting performance monitoring diagrams and setting are available at
    <https://my.sqreen.com/application/goto/settings/performance>.
    
    Note that the execution time monitoring diagram cannot be used as a strict
    Application Performance Monitoring diagram as it is based on a lossy exponential
    time-interval representation.
    Julio Guerra authored Oct 22, 2020
    Configuration menu
    Copy the full SHA
    e4f8db2 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2020

  1. agent/rule/callback/monitor-http-status-code: add not_found attack

    Log a `not_found` attack when the response is the NotFound status code 404. This
    attack is used by an internal backend playbook to detect vulnerability security
    scanners. The `HandleAttack()` API has been adapted so that callbacks can have
    more options and finer-grained control on the attack event creation.
    
    In order to avoid the cost of having multiple callbacks on the HTTP status code
    monitoring hookpoint (which would require the use of Go reflection), the
    not_found attack is added to the existing monitoring callback.
    Julio Guerra committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    a542038 View commit details
    Browse the repository at this point in the history
  2. agent/rule: adapt to the new HandleAttack API

    Julio Guerra committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    94a43cf View commit details
    Browse the repository at this point in the history
  3. agent/rule/callback/monitor-http-status-code: add tests

    Julio Guerra committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    65f548a View commit details
    Browse the repository at this point in the history
  4. agent: enhance the event manager event throughtput

    When dogfooding the not_found attack, we suddenly started having a
    high-frequency event. We therefore scaled the event queue to 10000 events by
    default, and the number of goroutines sending them to Sqreen can now scale up to
    the number of available CPUs. This allows to increase the overall throughput and
    handle more events rather than dropping them.
    Julio Guerra committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    6e98daa View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2020

  1. agent/rule/callback/monitor-http-status-code: add not_found events

    Log a `not_found` attack when the response is the NotFound status code 404. This
    attack is used by an internal backend playbook to detect vulnerability security
    scanners. The `HandleAttack()` API has been adapted so that callbacks can have
    more options and finer-grained control on the attack event creation.
    
    In order to avoid the cost of having multiple callbacks on the HTTP status code
    monitoring hookpoint (which would require the use of Go reflection), the
    not_found attack is added to the existing monitoring callback.
    
    When dogfooding the not_found attack, we suddenly started having a
    high-frequency event. We therefore scaled the event queue to 10000 events by
    default, and the number of goroutines sending them to Sqreen can now scale up to
    the number of available CPUs. This allows to increase the overall throughput and
    handle more events rather than dropping them.
    Julio Guerra authored Oct 29, 2020
    Configuration menu
    Copy the full SHA
    856c0a5 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2020

  1. sdk/middleware/echo: fix status code monitoring of handler errors

    Monitor the return error of Echo request handlers to see if they are Echo's
    HTTPErrors, so that we can properly capture the HTTP status code that will be
    later used by Echo. The HTTP response is indded out of the middleware scope in
    that case.
    Julio Guerra committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    7c8c77b View commit details
    Browse the repository at this point in the history
  2. sdk/middleware/echo: fix status code monitoring of handler errors

    Monitor the return error of Echo request handlers to see if they are Echo's
    HTTPErrors, so that we can properly capture the HTTP status code that will be
    later used by Echo. The HTTP response is otherwise out of the middleware scope in
    that case.
    Julio Guerra authored Oct 30, 2020
    Configuration menu
    Copy the full SHA
    344793d View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2020

  1. sqlib: error backoff logging

    Add backoff error logging for better error management in the request hotpaths. A
    backoff error logger is created using `plog.WithBackoff()`. It manages error
    counters and logs the error every time the counter value is a power of two.
    Error can be either individually counted when wrapped by `sqerrors.WithKey()`,
    or simply share the common counter of the backoff error logger object.
    Julio Guerra committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    bb740d2 View commit details
    Browse the repository at this point in the history
  2. agent: backoff error logger integration

    Simple backoff error logger integration in the agent and its callbacks. The
    error indexing should be further defined over time as it is currently
    oversampling due to shared keys (eg. avoiding using the same key for two
    distinct binding accessor expresssions).
    Julio Guerra committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    0b0339d View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2020

  1. agent: general fixes after overall dogfooding

    Julio Guerra committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    33c9148 View commit details
    Browse the repository at this point in the history
  2. agent: hot path error management

    Critical execution errors happening in the request hot path are now logged
    according to exponential backoff counters of their occurrences. This avoids
    slowing down request handlers but also spamming the internal agent queue.
    Such error will now go through the agent error logging facility every time the
    backoff counter is a power of two (1, 2, 4, 8, etc.). This is disabled when in
    debug log level.
    Julio Guerra authored Nov 3, 2020
    Configuration menu
    Copy the full SHA
    f6e3046 View commit details
    Browse the repository at this point in the history
  3. sdk/middleware/sqgin: fix response's content-lenght monitoring

    Correctly handle the default response case.
    Julio Guerra committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    be1a2c6 View commit details
    Browse the repository at this point in the history
  4. sdk/middleware: add response monitoring tests

    Julio Guerra committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    9a1e81d View commit details
    Browse the repository at this point in the history
  5. sdk/middleware/sqgin: response's content length monitoring

    Fix the response's content-length monitoring with the default response (ie. the handler does nothing).
    Julio Guerra authored Nov 3, 2020
    Configuration menu
    Copy the full SHA
    ce914a0 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2020

  1. agent/rule/callback/security-response: fix the event name and restore…

    … the tests
    
    Use the track event name the backend expects. Restoring the tests highlighted
    some inconsistent blocking behavior between user/ip blocking/redirecting that
    are now fixed.
    Julio Guerra committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    f6ddb94 View commit details
    Browse the repository at this point in the history
  2. agent/rule/callback/security-responses: fix the event name of securit…

    …y response actions
    
    Enable the correct event display on the dashboard by fixing the track event name with
    the one the backend actually expects.
    Also, restoring the security response tests highlighted some inconsistent blocking behavior
    between user/ip blocking/redirecting that are now fixed.
    Julio Guerra authored Nov 5, 2020
    Configuration menu
    Copy the full SHA
    ecb633a View commit details
    Browse the repository at this point in the history
  3. sdk: move to public interface API

    Moving to an interface-only public API has the benefit of hiding the
    type-implementation details that may change over time in the future. The
    returned SDK values are now interface values and no longer pointer values. This
    may break integrations having explicit types. We recommend to instead use
    type-inference when possible.
    
    From now, this change will allow us to transparently change the underlying
    values returned by SDK calls. For example, as of today, the underlying SDK
    values are small event collection wrappers that shouldn't be returned by
    reference in order to avoid useless allocations, for the benefit of the overall
    application performance. If the underlying object size increase in the future,
    we will be able to return allocated structures instead without breaking existing
    SDK integrations.
    Julio Guerra committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    7067c0b View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2020

  1. sdk: interface-only public sdk api

    Moving to an interface-only public API has the benefit of hiding the
    type-implementation details that may change over time in the future. The
    returned SDK values are now interface values and no longer pointer values. This
    may break integrations having explicit types. We recommend to instead use
    type-inference when possible.
    
    From now, this change will allow us to transparently change the underlying
    values returned by SDK calls. For example, as of today, the underlying SDK
    values are small event collection wrappers that shouldn't be returned by
    reference in order to avoid useless allocations, for the benefit of the overall
    application performance. If the underlying object size increase in the future,
    we will be able to return allocated structures instead without breaking existing
    SDK integrations.
    Julio Guerra authored Nov 6, 2020
    Configuration menu
    Copy the full SHA
    a2837c6 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2020

  1. agent/rule/callback/binding-accessors/sql: unwrap the SQL driver befo…

    …re reading its package path
    
    Following up on elastic/apm-agent-go#848 and the
    addition of the `Unwrap()` method to Elastic's SQL driver tracer, it is now
    possible to unwrap it and properly read the package path of the underlying
    driver as expected.
    Julio Guerra committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    9ff0c80 View commit details
    Browse the repository at this point in the history
  2. agent/rule/callback/binding-accessors/sql: finer-grained SQL dialect …

    …method errors
    
    Add finer-grained error keys that can be returned by the SQL dialect getter
    method. This also highlighted the fact errors can be assigned multiple keys
    while bubbling up the call stack. Therefore, the quick solution to get an error
    key is to make `sqerrors.Key()` return the deepest key rather than the top one.
    Ideally, it should rather combine multiple keys together in the future.
    Julio Guerra committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    204c448 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2020

  1. agent/rule/callback/binding-accessors/sql: unwrap the SQL driver befo…

    …re reading its package path
    
    Following up on elastic/apm-agent-go#848 and the addition of the `Unwrap()`
    method to Elastic's SQL driver tracer, it is now possible to unwrap it and properly
    read the package path of the underlying driver as expected.
    Julio Guerra authored Nov 12, 2020
    Configuration menu
    Copy the full SHA
    f87a508 View commit details
    Browse the repository at this point in the history
  2. sdk/middleware/sqhttp: transparent response writer wrapper

    Make the HTTP response writer wrapper transparent by implementing the same
    *known* interfaces as the underlying HTTP response writer it implements. The
    list of interfaces is currently every optional `net/http` interfaces, and some
    from `io` when relevant:
    
    	- `http.Flusher`: to allow flushing any buffered to the client. This enables
    	  support for streaming handlers.
    
    	- `http.Hijacker`: to allow handlers to takeover the HTTP connection. This
    	  should enable the support for websocket servers, which are not officially
    	  supported by Sqreen, but is now experimentally allowed.
    
    	- `http.Pusher`: for HTTP2 server push.
    
    	- `http.CloseNotifier`: the deprecated closed connection notifier.
    
    	- `io.ReaderFrom`: for optimized copies (eg. `io.Copy(file, w)`)
    
    	- `io.WriteString`: for optimized string write (which avoids a temporary string copy into a byte slice)
    
    The transparent wrapper implementation has been generated from a tool that will
    be released in the Go agent repository in the future.
    
    Fixes #162 and #134
    Julio Guerra committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    b8d555a View commit details
    Browse the repository at this point in the history
  3. sdk/middleware/{sqgin,sqecho}: response writer wrapper cleanup

    Remove the useless response writer wrapper as Echo and Gin already monitor them.
    Note that Gin actually doesn't allow changing the writter and all its helpers
    use the internal in-memory writer. Therefore, we are for now accepting multiple
    responses to keep the original response writer behaviour.
    Julio Guerra committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    6f3abd2 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2020

  1. agent/protection/http/binding-accessor: unalias url.Values to fix the…

    … js conversion
    Julio Guerra committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    58eeeae View commit details
    Browse the repository at this point in the history
  2. sdk/middleware/sqhttp: transparent http response writer wrapper

    Make the HTTP response writer wrapper transparent by implementing the same
    *known* interfaces as the underlying HTTP response writer. The
    list of interfaces is currently every optional `net/http` interfaces, and some
    from `io` when relevant:
    
      - `http.Flusher`: to allow flushing any buffered to the client. This enables
        support for streaming handlers.
      - `http.Hijacker`: to allow handlers to takeover the HTTP connection. This
    	  should enable the support for websocket servers, which are not officially
    	  supported by Sqreen, but is now experimentally allowed.
      - `http.Pusher`: for HTTP2 server push.
      - `http.CloseNotifier`: the deprecated closed connection notifier.
      - `io.ReaderFrom`: for optimized copies (eg. `io.Copy(file, w)`)
      - `io.WriteString`: for optimized string write (which avoids a temporary string copy into a byte slice)
    
    The transparent wrapper implementation has been generated with a tool that will
    be released in the Go agent repository in the future.
    
    Fixes #162 and #134
    Julio Guerra authored Nov 17, 2020
    Configuration menu
    Copy the full SHA
    b4824cc View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2020

  1. agent/metrics: fix metrics regressions

    - Restore max length checks
    - Update max length error checks
    Julio Guerra authored Nov 18, 2020
    Configuration menu
    Copy the full SHA
    9b0856e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d4c64f0 View commit details
    Browse the repository at this point in the history
  3. agent: fix the early logger log level

    Julio Guerra committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    a0d8208 View commit details
    Browse the repository at this point in the history
  4. sdk: add FromRequest() function helper

    Julio Guerra committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    402e454 View commit details
    Browse the repository at this point in the history
  5. sdk: add FromRequest() helper

    Add `sdk.FromRequest()` helper function to retrieve Sqreen's SDK handle from a request.
    It is a shortcut to `sdk.FromContext(r.Context())`.
    Julio Guerra authored Nov 18, 2020
    Configuration menu
    Copy the full SHA
    ded1fbb View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2020

  1. repo: add v1.0.0 changelog

    Julio Guerra committed Nov 19, 2020
    Configuration menu
    Copy the full SHA
    31ce2bb View commit details
    Browse the repository at this point in the history