Add ability to configure pg error_handler

[alexevanczuk]

What does this PR do?

This PR allows the client to configure an error_handler for the pg integration, which does not currently support an error_handler as many of the other integrations.

Motivation:

Application level rescuing

Take the following ruby code snippet:

begin
  insert_non_unique_record
rescue ActiveRecord::RecordNotUnique => e
  # We catch this error and handle it gracefully
end

In this example, the pg integration reports an PG::UniqueViolation error to datadog. This error is pure noise, because our application code catches the exception generated by the postgres ruby gem. Since we're rescuing this, it conceptually makes sense to also not report the postgres error.

Application level handling of all errors

Consider this code snippet:

insert_non_unique_record

Even though this is not rescued as above, the PG::UniqueViolation will propagate through the stack and eventually be caught by the overall Datadog middleware for web or async transactions and be reported to Datadog. To avoid double reporting of errors, we'd like to ignore these.

Console level mistakes

Consider I'm in console and I make an error in my postgres query. This will be reported to datadog. I might want to ignore errors from console or just ignore all errors unless other parts of the stack rescue errors in console and report or log them.

Additional Notes

Let me know if there's other best practices for implementing this change.

For what it's worth – I also tried to ignore this through some other various hacks.

I tried this:

Datadog::Tracing.before_flush do |trace|
  trace.spans.each do |span|
    # See note above about service naming.
    next if span.get_tag('service') != 'pg'
    span.clear_tag('error')
  end

  trace
end

But this did not work. It looked like it did not actually clear the error span tag or the error status from the trace.

I'd love some other way to be able to more forcibly override any integration to remove errors from certain spans where they do not apply, but it does not appear there is a public API to do this yet.

How to test the change?

I followed patterns in the faraday/middleware_spec.rb to add tests to cover this.

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[alexevanczuk]

Please let me know if there is another standard way to get the error handler here.

[alexevanczuk]

Best I can tell this is the literal constant that represented the existing default behavior, so I wanted to reference it explicitly here to maintain existing behavior.

[urseberry]

Minor grammar nit, otherwise approved.

[alexevanczuk]

Merged your suggestion, thanks @urseberry ! Let me know if there's anything else I can help with to get this merged and released. Thanks for your prompt review!

[alexevanczuk]

Hey @urseberry wanted to bump this one! Let me know if there's anything else ya need 🙏🏼

[urseberry]

Hey @urseberry wanted to bump this one! Let me know if there's anything else ya need 🙏🏼

Thanks, Alex! I already approved on behalf of Documentation, but you also need an engineer to review and approve. It's in their queue and I'm sure someone will get to your PR soon.

[alexevanczuk]

Thanks @marcotc for the merge! No rush, but just curious to have a sense of release cadence!

[ekump]

@alexevanczuk We released 1.18.0 today and this has been included. Thank you for your contribution!

[alexevanczuk]

Thanks so much @ekump !