[pgbouncer] Fix stats deprecated in pgbouncer 1.8 #1016

sj26 · 2018-01-16T04:34:58Z

When using pgbouncer version 1.8 the pgbouncer check fails with:

$ datadog-agent check pgbouncer
  =========
  Collector
  =========

    Running Checks
    ==============
      pgbouncer
      ---------
        Total Runs: 1
        Metrics: 0, Total Metrics: 0
        Events: 0, Total Events: 0
        Service Checks: 0, Total Service Checks: 0Error:
        Traceback (most recent call last):
          File "/opt/datadog-agent/bin/agent/dist/checks/__init__.py", line 300, in run
            self.check(copy.deepcopy(self.instances[0]))
          File "/opt/datadog-agent/checks.d/pgbouncer.py", line 227, in check
            self._collect_stats(db, tags)
          File "/opt/datadog-agent/checks.d/pgbouncer.py", line 121, in _collect_stats
            assert len(row) == len(cols) + len(desc)
        AssertionError

This is due to some stats being renamed in an upstream commit as part of pgbouncer 1.8: pgbouncer/pgbouncer@876d8a5

The attached commit updates the datadog pgbouncer check a little, adds a test flavor version for pgbouncer 1.8, and modifies the check to cope with the renamed and added stats maintaining backwards compatibility.

(We're now running this version of the check in production.)

I didn't see this work had already been started in #1011 — feel free to take this commit and fold it in there, or whatever you'd like. (I've kept version and changelog updates out of here so it can be cherried or rebased.)

Adds new column support while maintaining backward compat, detected using result set column names. See pgbouncer/pgbouncer@876d8a5

nmuesch

Hey, thanks for the PR! (Also appreciate the tests!) Left a couple of quick thoughts as well, let me know what you think.

nmuesch · 2018-01-17T17:01:15Z

pgbouncer/datadog_checks/pgbouncer/pgbouncer.py


-                    desc = scope['descriptors']
+                    try:
+                        self.log.debug("Running query: %s" % query)


It may be helpful to have a debug line here that also reports the output of the query to help troubleshoot possible issues in the future. What do you think?

I did add something like this but was worried it might be too much output to the log so omitted from the final commit. If you think it's a good idea I'll add it back.

nmuesch · 2018-01-17T17:02:58Z

pgbouncer/test/test_pgbouncer.py

@@ -22,6 +23,9 @@ class TestPgbouncer(AgentCheckTest):
    CHECK_NAME = 'pgbouncer'

    def test_checks(self):
+        pgbouncer_version = os.environ.get('FLAVOR_VERSION', 'latest')
+        pgbouncer_deprecated = pgbouncer_version in ('1.5', '1.7')


Thanks for writing the tests! I'm not sure we want to deprecate older versions of PGBouncer. Could we rename this to something like 'pgbouncer_pre18` or something similar here?

Good idea! I was trying to use something other than "old" and "new" 😅

sj26 · 2018-01-18T01:28:16Z

(Separated commits for review purposes, probably should be squashed for merge.)

hush-hush

Hi @sj26,

Thanks a lot for this amazing PR ! I added a few comment but nothing blocking.

Could you also update the CHANGELOG (creating a new Unreleased section) and the manifest.json and postgres/datadog_checks/postgres/__init__.py to update the check version. You can find a exemple here

hush-hush · 2018-01-18T16:58:13Z

pgbouncer/datadog_checks/pgbouncer/pgbouncer.py

-                    if row[0] == self.DB_NAME:
-                        continue
+                    except pg.Error as e:
+                        self.log.exception("Not all metrics may be available")


I think we should still log the PG exception, it might be useful. What do you think ?

Using log.exception will print the exception details in the log per the docs, e.g.:

checks.pgbouncer: ERROR: Not all metrics may be available Traceback (most recent call last): File "/Users/sj26/Projects/datadog/integrations-core/pgbouncer/datadog_checks/pgbouncer/pgbouncer.py", line 106, in _collect_stats cursor.execute(query) File "/Users/sj26/Projects/datadog/integrations-core/venv/lib/python2.7/site-packages/psycopg2/extras.py", line 144, in execute return super(DictCursor, self).execute(query, vars) ProgrammingError: unrecognized configuration parameter "pools"

indeed, thanks !

hush-hush · 2018-01-18T18:48:17Z

pgbouncer/datadog_checks/pgbouncer/pgbouncer.py


-                    assert len(row) == len(cols) + len(desc)
+                            tags = list(instance_tags)
+                            tags += ["%s:%s" % (tag, row[column]) for (column, tag) in descriptors]


We should test if row contains column to avoid raising an exception if the column in descriptor does not exist.
Something like:

tags += ["%s:%s" % (tag, row[column]) for (column, tag) in descriptors if column in row]

These are pretty important and should never be missing. If they are missing then it should probably explode.

I'll add the conditional anyway as protection against column name changes 👍

hush-hush · 2018-01-18T18:52:01Z

pgbouncer/datadog_checks/pgbouncer/pgbouncer.py

            ('total_received',       ('pgbouncer.stats.bytes_received_per_second', RATE)),
            ('total_sent',           ('pgbouncer.stats.bytes_sent_per_second', RATE)),
            ('total_query_time',     ('pgbouncer.stats.total_query_time', GAUGE)),
-            ('avg_req',              ('pgbouncer.stats.avg_req', GAUGE)),
+            ('total_xact_time',      ('pgbouncer.stats.total_transaction_time', GAUGE)), # >= 1.8


I think this should be a rate since the documentation say:

Total number of microseconds spent by **pgbouncer** when connected to PostgreSQL

I was copying total_query_time — if this should change, so should that. Probably makes sense that all the "total" are rates, and all the "avg" are gauges. Will update.

hush-hush · 2018-01-18T19:14:55Z

pgbouncer/datadog_checks/pgbouncer/pgbouncer.py

        except pg.Error as e:
-            self.log.error("Connection error: %s" % str(e))
+            self.log.exception("Connection error")


Also e is never used which make the flake8 fail (that's why the travis tests are red).

I'll remove!

hush-hush · 2018-01-19T16:06:00Z

Thanks a lot for this PR @sj26 !

It will be available in the next version of the agent 5.22.0.

[pgbouncer] Fix stats deprecated in 1.8

2266817

Adds new column support while maintaining backward compat, detected using result set column names. See pgbouncer/pgbouncer@876d8a5

hush-hush assigned masci and hush-hush Jan 16, 2018

hush-hush added the community label Jan 16, 2018

nmuesch mentioned this pull request Jan 17, 2018

[PGBouncer] Adds support for 1.8 new metrics #1011

Closed

3 tasks

nmuesch reviewed Jan 17, 2018

View reviewed changes

sj26 added 2 commits January 18, 2018 12:16

Avoid "deprecated"

2710971

Brush up logging

e63315d

hush-hush unassigned masci Jan 18, 2018

hush-hush requested changes Jan 18, 2018

View reviewed changes

sj26 added 4 commits January 19, 2018 14:37

We never use e

67dd765

These should be rates not gauges

419f606

Conditionally set descriptors in case column names change

ed7d459

Update version and changelog

8bfef1f

hush-hush approved these changes Jan 19, 2018

View reviewed changes

hush-hush added this to the 5.22 milestone Jan 19, 2018

hush-hush merged commit cabaec2 into DataDog:master Jan 19, 2018

jdreaver mentioned this pull request Jan 25, 2018

[core] PGBouncer stats for version latest fails due to extra columns #555

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pgbouncer] Fix stats deprecated in pgbouncer 1.8 #1016

[pgbouncer] Fix stats deprecated in pgbouncer 1.8 #1016

sj26 commented Jan 16, 2018

nmuesch left a comment

nmuesch Jan 17, 2018

sj26 Jan 18, 2018

nmuesch Jan 17, 2018

sj26 Jan 18, 2018

sj26 commented Jan 18, 2018 •

edited

Loading

hush-hush left a comment •

edited

Loading

hush-hush Jan 18, 2018

sj26 Jan 19, 2018

hush-hush Jan 19, 2018

hush-hush Jan 18, 2018

sj26 Jan 19, 2018

sj26 Jan 19, 2018

hush-hush Jan 18, 2018

sj26 Jan 18, 2018

hush-hush Jan 18, 2018

sj26 Jan 19, 2018

hush-hush commented Jan 19, 2018

[pgbouncer] Fix stats deprecated in pgbouncer 1.8 #1016

[pgbouncer] Fix stats deprecated in pgbouncer 1.8 #1016

Conversation

sj26 commented Jan 16, 2018

nmuesch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sj26 commented Jan 18, 2018 • edited Loading

hush-hush left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hush-hush commented Jan 19, 2018

sj26 commented Jan 18, 2018 •

edited

Loading

hush-hush left a comment •

edited

Loading