feat: Add customizable log redactor that can handle both Hash and String payload. Also add support for auto-generating request IDs

README.md

@@ -6,6 +6,20 @@ The goal of this library is to add a layer of abstraction on top of [Faraday](ht
 
 ## Usage
 
+**Configuration:**
+```ruby
+HTTPigeon.configure do |config|
+  config.default_event_type = # Set a default event type for all requests, overridable per request. Default: 'http.outbound'
+  config.default_filter_keys = # Set a default list of keys to be redacted for Hash payloads, overridable per request. Default: []
+  config.redactor_string = # Set a string that should be used as the replacement when redacting sensitive data. Default: '[FILTERED]'
+  config.log_redactor = # Specify an object to be used for redacting data before logging. Must respond to #redact(data<Hash, String>). Default: nil
+  config.event_logger = # Specify an object to be used for logging request roundtrip events. Default: $stdout
+  config.auto_generate_request_id = # Auto-generate a uuid for each request and store in a 'X-Request-Id' header?
+  config.exception_notifier = # Specify an object to be used for reporting errors. Must respond to #notify_exception(e<Exception>)
+  config.notify_all_exceptions = # Do you want these errors to actually get reported/notified?
+end
+```
+
 **Instantiating with a block:**
 ```ruby
 # @option [String] base_url the base URI (required)
@@ -24,14 +38,28 @@ request.run(path: '/users/1')
 
 **Instantiating with customizable arguments:**
 ```ruby
-# @option [String] base_url the base URI (required)
-# @option [Hash] options the Faraday connection options (default: {})
-# @option [Hash] headers the request headers (default: {})
-# @option [Faraday::Adapter] adapter the Faraday adapter (default: Net::HTTP)
-# @option [Logger] logger for logging request and response (default: HTTPigeon::Logger)
-# @option [String] event_type for filtering/scoping the logs (default: 'http.outbound')
-# @option [Array] filter_keys list of keys in headers and body to be redacted before logging (default: [])
-request = HTTPigeon::Request.new(base_url: 'https://dummyjson.com', headers: { Accept: 'application/json' }, filter_keys: [:ssn, :password])
+# @param type [Symbol, String] the type of object this filter will be applied to (:hash or :string)
+# @param pattern [Symbol, String, Regex] the exact key or pattern that should be redacted
+# @param sub_prefix [String] a prefix to be combined with the configured :redactor_string as the replacement for the sensitive data (default: nil)
+# @param replacement [String] a string to be used as the replacement for the sensitive data.
+#    If :sub_prefix is defined, this value will be ignored (default: nil)
+filter_1 = HTTPigeon::Filter.new(:hash, 'access_token')
+filter_2 = HTTPigeon::Filter.new(:string, /username=[0-9a-z]*/i, 'username=')
+filter_3 = HTTPigeon::Filter.new(:string, /password=[0-9a-z]*/i, nil, 'password=***')
+
+# @param base_url [String] the base URI
+# @param options [Hash] the Faraday connection options (default: {})
+# @param headers [Hash] the request headers (default: {})
+# @param adapter [Faraday::Adapter] the Faraday adapter (default: Net::HTTP)
+# @param logger [Logger] for logging request and response (default: HTTPigeon::Logger)
+# @param event_type [String] for filtering/scoping the logs (default: 'http.outbound')
+# @param filter_keys [Array<String, Symbol>] specifies keys in headers and body to be redacted before logging.
+#    Can only define keys for Hash payloads (default: [])
+# @param log_filters [Array<HTTPigeon::Filter, Object>] specifies keys in headers and body to be redacted before logging.
+#    Can define keys for both Hash and String payloads (default: [])
+# @note :filter_keys and :log_filters can both be specified but it is recommended to define all filters using :log_filters
+#    if you wish to filter both Hashes and Strings
+request = HTTPigeon::Request.new(base_url: 'https://dummyjson.com', headers: { Accept: 'application/json' }, filter_keys: [:ssn, :ip_address], log_filters: [filter_1, filter_2, filter_3])
 request.run(path: '/users/1')
 ```
 
@@ -60,10 +88,12 @@ class CustomLogger < Logger
     super(:info, log_data.to_json)
   end
 
+  # optional method
   def on_request_start
     @start_time = Time.current
   end
 
+  # optional method
   def on_request_finish
     @end_time = Time.current
   end
@@ -75,7 +105,7 @@ request.run(path: '/users/1')
 
 **Using the default logger:**
 
-To use the default logger (`HTTPigeon::Logger`), simply pass a custom `:filter_keys` and `:event_type` args, if necessary, and you're all set.
+To use the default logger (`HTTPigeon::Logger`), simply pass a custom `:filter_keys`, `:log_filters` and `:event_type` args, if necessary, and you're all set.
 
 **Running a request:**
 

httpigeon.gemspec

@@ -7,7 +7,7 @@ Gem::Specification.new do |spec|
   spec.version = HTTPigeon::VERSION
   spec.authors = ["2k-joker"]
   spec.email = ["[email protected]"]
-  spec.licenses = ["Nonstandard"]
+  spec.licenses = ["MIT"]
 
   spec.summary = "Simple, easy way to make and log HTTP requests and responses"
   spec.description = "Client library that simplifies making and logging HTTP requests and responses. This library is built as an abstraction on top of the Faraday ruby client."

lib/httpigeon.rb

@@ -2,13 +2,23 @@
 
 require "httpigeon/configuration"
 require "httpigeon/version"
+require "httpigeon/filter"
+require "httpigeon/log_redactor"
 require "httpigeon/logger"
 require "httpigeon/request"
 
 module HTTPigeon
   extend self
 
-  delegate :default_event_type, :default_filter_keys, :event_logger, :notify_all_exceptions, :exception_notifier, to: :configuration
+  delegate :default_event_type,
+           :default_filter_keys,
+           :redactor_string,
+           :log_redactor,
+           :event_logger,
+           :auto_generate_request_id,
+           :notify_all_exceptions,
+           :exception_notifier,
+           to: :configuration
 
   def configure
     @config = HTTPigeon::Configuration.new

lib/httpigeon/configuration.rb

@@ -1,11 +1,14 @@
 module HTTPigeon
   class Configuration
-    attr_accessor :default_event_type, :default_filter_keys, :event_logger, :notify_all_exceptions, :exception_notifier
+    attr_accessor :default_event_type, :default_filter_keys, :redactor_string, :log_redactor, :event_logger, :notify_all_exceptions, :exception_notifier, :auto_generate_request_id
 
     def initialize
       @default_event_type = 'http.outbound'
       @default_filter_keys = []
+      @redactor_string = '[FILTERED]'
+      @log_redactor = nil
       @event_logger = nil
+      @auto_generate_request_id = false
       @notify_all_exceptions = false
       @exception_notifier = nil
     end

lib/httpigeon/filter.rb

@@ -0,0 +1,12 @@
+module HTTPigeon
+  class Filter
+    attr_reader :type, :pattern, :sub_prefix, :replacement
+
+    def initialize(type, pattern, sub_prefix = nil, replacement = nil)
+      @type = type
+      @pattern = pattern
+      @sub_prefix = sub_prefix
+      @replacement = replacement
+    end
+  end
+end

lib/httpigeon/log_redactor.rb

@@ -0,0 +1,51 @@
+require "active_support/core_ext/hash"
+
+module HTTPigeon
+  class LogRedactor
+    attr_reader :hash_filter_keys, :string_filters
+
+    def initialize(hash_filter_keys: nil, string_filters: nil)
+      @hash_filter_keys = hash_filter_keys.map(&:to_s).map(&:downcase)
+      @string_filters = string_filters || []
+    end
+
+    def redact(data)
+      case data
+      when Array
+        data.map { |datum| redact(datum) }
+      when String
+        redact_string(data)
+      when Hash
+        redact_hash(data)
+      else
+        data
+      end
+    end
+
+    private
+
+    def redact_hash(data)
+      data.to_h do |k, v|
+        v = HTTPigeon.redactor_string if hash_filter_keys.include?(k.to_s.downcase)
+
+        if v.is_a?(Hash) || v.is_a?(Array)
+          [k, redact_hash(v)]
+        else
+          [k, v]
+        end
+      end
+    end
+
+    def redact_string(data)
+      string_filters.each do |filter|
+        data = if filter.sub_prefix.present?
+                 data.gsub(filter.pattern, "#{filter.sub_prefix}#{HTTPigeon.redactor_string}")
+               else
+                 data.gsub(filter.pattern, filter.replacement)
+               end
+      end
+
+      data
+    end
+  end
+end

lib/httpigeon/logger.rb

@@ -3,16 +3,21 @@
 
 module HTTPigeon
   class Logger
-    def initialize(event_type: nil, additional_filter_keys: nil)
+    attr_reader :event_type, :log_redactor, :start_time, :end_time
+
+    def initialize(event_type: nil, additional_filter_keys: nil, log_filters: nil)
       @event_type = event_type || HTTPigeon.default_event_type
-      @additional_filter_keys = additional_filter_keys.to_a.map(&:to_s)
+
+      hash_filters, string_filters = (log_filters || []).partition { |filter| filter.type.to_sym == :hash }
+      hash_filter_keys = HTTPigeon.default_filter_keys + additional_filter_keys.to_a + hash_filters.map(&:pattern)
+      @log_redactor = HTTPigeon.log_redactor || HTTPigeon::LogRedactor.new(hash_filter_keys: hash_filter_keys, string_filters: string_filters)
     end
 
     def log(faraday_env, data = {})
       base_log_data = { event_type: event_type }
-      log_data = build_log_data(faraday_env, data)
+      log_data = build_log_data(faraday_env, data).merge(base_log_data)
 
-      HTTPigeon.event_logger.nil? ? log_to_stdout : HTTPigeon.event_logger.log(log_data.merge(base_log_data))
+      HTTPigeon.event_logger.nil? ? log_to_stdout(log_data) : HTTPigeon.event_logger.log(log_data)
     rescue StandardError => e
       HTTPigeon.exception_notifier.notify_exception(e) if HTTPigeon.notify_all_exceptions
       raise e if ['development', 'test'].include?(ENV['RAILS_ENV'].to_s)
@@ -28,8 +33,6 @@ def on_request_finish
 
     private
 
-    attr_reader :event_type, :header_log_keys, :additional_filter_keys, :start_time, :end_time
-
     def build_log_data(env, data)
       log_data = data.deep_dup
       request_id = env.request_headers.transform_keys(&:downcase)['x-request-id']
@@ -68,39 +71,17 @@ def build_log_data(env, data)
       log_data
     end
 
-    def filter_keys
-      @filter_keys ||= (HTTPigeon.default_filter_keys + additional_filter_keys).map(&:downcase)
-    end
-
     def filter(body)
       return {} if body.blank?
 
       body = JSON.parse(body) if body.is_a?(String)
-      filter_hash(body)
+      log_redactor.redact(body)
     rescue JSON::ParserError
-      body
-    end
-
-    def filter_hash(data)
-      if data.is_a?(Array)
-        data.map { |datum| filter_hash(datum) }
-      elsif !data.is_a?(Hash)
-        data
-      else
-        data.to_h do |k, v|
-          v = '[FILTERED]' if filter_keys.include?(k.to_s.downcase)
-
-          if v.is_a?(Hash) || v.is_a?(Array)
-            [k, filter_hash(v)]
-          else
-            [k, v]
-          end
-        end
-      end
+      log_redactor.redact(body)
     end
 
-    def log_to_stdout
-      Logger.new($stdout).log(1, { event_type: event_type, data: log_data }.to_json)
+    def log_to_stdout(log_data)
+      Logger.new($stdout).log(1, log_data.to_json)
     end
   end
 end

lib/httpigeon/request.rb

@@ -10,10 +10,11 @@ class Request
 
     delegate :status, :body, to: :response, prefix: true
 
-    def initialize(base_url:, options: nil, headers: nil, adapter: nil, logger: nil, event_type: nil, filter_keys: nil)
+    def initialize(base_url:, options: nil, headers: nil, adapter: nil, logger: nil, event_type: nil, filter_keys: nil, log_filters: nil)
       @base_url = base_url
       @event_type = event_type
       @filter_keys = filter_keys || []
+      @log_filters = log_filters
       @logger = logger || default_logger
 
       request_headers = default_headers.merge(headers.to_h)
@@ -47,7 +48,7 @@ def run(method: :get, path: '/', payload: {})
 
     private
 
-    attr_reader :path, :logger, :event_type, :filter_keys
+    attr_reader :path, :logger, :event_type, :filter_keys, :log_filters
 
     def parse_response
       JSON.parse(response_body).with_indifferent_access unless response_body.empty?
@@ -56,11 +57,11 @@ def parse_response
     end
 
     def default_logger
-      HTTPigeon::Logger.new(event_type: event_type, additional_filter_keys: filter_keys)
+      HTTPigeon::Logger.new(event_type: event_type, additional_filter_keys: filter_keys, log_filters: log_filters)
     end
 
     def default_headers
-      { 'Accept' => 'application/json' }
+      HTTPigeon.auto_generate_request_id ? { 'Accept' => 'application/json', 'X-Request-Id' => SecureRandom.uuid } : { 'Accept' => 'application/json' }
     end
   end
 end

test/httpigeon/log_redactor_test.rb

@@ -0,0 +1,56 @@
+require_relative "../test_helper"
+
+class HTTPigeon::LogRedactorTest < HTTPigeon::TestCase
+  describe '#redact' do
+    let(:string_filters) { [HTTPigeon::Filter.new(:string, /key_3=[0-9a-z]*/i, 'key_3=')] }
+    let(:data) { nil }
+
+    let(:redactor) do
+      HTTPigeon::LogRedactor.new(hash_filter_keys: [:key_1, :key_2], string_filters: string_filters)
+    end
+
+    context 'when data is an array' do
+      let(:data) { [1, 'abc-123&key_3=supersecret007&xyz-789', { key_1: 'duper-secret', key_4: 'public-knowledge' }] }
+
+      it 'filters all elements' do
+        HTTPigeon.stub(:redactor_string, '<redacted>') do
+          assert_equal [1, 'abc-123&key_3=<redacted>&xyz-789', { key_1: '<redacted>', key_4: 'public-knowledge' }], redactor.redact(data)
+        end
+      end
+    end
+
+    context 'when data is a string' do
+      let(:data) { 'abc-123&key_3=supersecret007&xyz-789' }
+
+      context 'when a :sub_prefix is defined' do
+        it 'filters the string' do
+          HTTPigeon.stub(:redactor_string, '<redacted>') do
+            assert_equal 'abc-123&key_3=<redacted>&xyz-789', redactor.redact(data)
+          end
+        end
+      end
+
+      context 'when a :sub_prefix is not defined' do
+        let(:string_filters) { [HTTPigeon::Filter.new(:string, /key_3=[0-9a-z]*/i, nil, 'key_3=***')] }
+
+        it 'filters the string' do
+          HTTPigeon.stub(:redactor_string, '<redacted>') do
+            assert_equal 'abc-123&key_3=***&xyz-789', redactor.redact(data)
+          end
+        end
+      end
+    end
+
+    context 'when data is a hash' do
+      let(:data) { { key_1: 'duper-secret', key_4: 'public-knowledge', key_5: { key_2: 'duper-duper-secret' } } }
+
+      it 'filters all elements' do
+        HTTPigeon.stub(:redactor_string, '<redacted>') do
+          expected = { key_1: '<redacted>', key_4: 'public-knowledge', key_5: { key_2: '<redacted>' } }
+
+          assert_equal expected, redactor.redact(data)
+        end
+      end
+    end
+  end
+end