Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DBMON-2435] serialize payload with type bytes in json dumps #15763

Merged
merged 11 commits into from
Sep 8, 2023

Conversation

lu-zhengda
Copy link
Contributor

@lu-zhengda lu-zhengda commented Sep 5, 2023

What does this PR do?

This PR fixes JSON serialization bug TypeError: Type is not JSON serializable: bytes in SQL Server activity collection, which cause the check to abruptly shutdown and restart.

Motivation

Fixing a long standing bug reported by customer since v7.37.1 which causes the SQL Server activity collection to shutdown and restart.

Traceback (most recent call last):
  File "C:\Program Files\Datadog\Datadog Agent\embedded3\lib\site-packages\datadog_checks\base\utils\tracking.py", line 71, in wrapper
    result = function(self, *args, **kwargs)
  File "C:\Program Files\Datadog\Datadog Agent\embedded3\lib\site-packages\datadog_checks\sqlserver\activity.py", line 303, in collect_activity
    payload = json.dumps(event, default=default_json_event_encoding)
TypeError: Type is not JSON serializable: bytes

Additional Notes

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Changelog entries must be created for modifications to shipped code
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.

@ghost ghost added the documentation label Sep 5, 2023
@codecov
Copy link

codecov bot commented Sep 5, 2023

Codecov Report

Merging #15763 (eb058ff) into master (1d70f94) will increase coverage by 0.18%.
The diff coverage is 87.50%.

Flag Coverage Δ
active_directory 100.00% <ø> (+17.64%) ⬆️
activemq 52.87% <ø> (ø)
activemq_xml 82.31% <ø> (ø)
amazon_msk 89.07% <ø> (ø)
ambari 85.75% <ø> (ø)
apache 95.08% <ø> (ø)
arangodb 98.23% <ø> (ø)
argocd 88.04% <ø> (ø)
aspdotnet 100.00% <ø> (ø)
avi_vantage 91.35% <ø> (ø)
azure_iot_edge 82.08% <ø> (ø)
boundary 100.00% <ø> (ø)
btrfs 82.91% <ø> (ø)
cacti 87.90% <ø> (ø)
calico 84.61% <ø> (ø)
cassandra 65.51% <ø> (ø)
cert_manager 77.41% <ø> (ø)
cilium 77.63% <ø> (+0.87%) ⬆️
cisco_aci 95.27% <ø> (ø)
citrix_hypervisor 87.50% <ø> (ø)
cloud_foundry_api 96.35% <ø> (+0.12%) ⬆️
cloudera 99.49% <ø> (ø)
cockroachdb 91.52% <ø> (ø)
consul 91.65% <ø> (ø)
coredns 94.57% <ø> (ø)
couch 95.43% <ø> (+0.24%) ⬆️
crio 89.79% <ø> (ø)
datadog_checks_base 89.67% <87.50%> (+0.39%) ⬆️
datadog_checks_dev 82.51% <ø> (+0.07%) ⬆️
datadog_checks_downloader 81.65% <ø> (ø)
datadog_cluster_agent 90.19% <ø> (ø)
dcgm 97.40% <ø> (ø)
ddev 99.63% <ø> (ø)
directory 95.87% <ø> (+0.65%) ⬆️
disk 89.23% <ø> (ø)
dns_check 93.90% <ø> (ø)
dotnetclr 91.39% <ø> (+12.90%) ⬆️
druid 98.47% <ø> (ø)
ecs_fargate 82.91% <ø> (ø)
eks_fargate 94.05% <ø> (ø)
envoy 95.09% <ø> (+0.41%) ⬆️
etcd 95.56% <ø> (ø)
exchange_server 96.85% <ø> (+11.81%) ⬆️
external_dns 89.28% <ø> (ø)
fluentd 94.77% <ø> (ø)
foundationdb 78.50% <ø> (ø)
gearmand 78.26% <ø> (+1.24%) ⬆️
gitlab_runner 91.94% <ø> (ø)
go_expvar 92.73% <ø> (ø)
gunicorn 92.85% <ø> (+0.75%) ⬆️
harbor 89.68% <ø> (ø)
hazelcast 92.39% <ø> (ø)
hdfs_datanode 89.74% <ø> (ø)
hdfs_namenode 86.72% <ø> (ø)
hive 51.42% <ø> (ø)
hivemq 61.90% <ø> (ø)
http_check 95.51% <ø> (+2.05%) ⬆️
hudi 73.91% <ø> (ø)
ibm_ace 91.89% <ø> (ø)
ibm_db2 95.30% <ø> (ø)
ibm_i 81.91% <ø> (ø)
ibm_mq 91.40% <ø> (ø)
ibm_was 96.08% <ø> (ø)
ignite 46.66% <ø> (ø)
impala 97.97% <ø> (ø)
istio 76.45% <ø> (+0.55%) ⬆️
jboss_wildfly 47.36% <ø> (ø)
kafka 63.63% <ø> (ø)
kong 87.56% <ø> (ø)
kube_apiserver_metrics 97.85% <ø> (ø)
kube_controller_manager 96.07% <ø> (ø)
kube_dns 95.97% <ø> (ø)
kube_metrics_server 94.87% <ø> (ø)
kube_proxy 96.80% <ø> (ø)
kube_scheduler 97.50% <ø> (ø)
kubelet 91.03% <ø> (ø)
kubernetes_state 89.18% <ø> (ø)
kyototycoon 85.96% <ø> (ø)
lighttpd 83.64% <ø> (ø)
linkerd 85.14% <ø> (+1.14%) ⬆️
linux_proc_extras 96.22% <ø> (ø)
mapr 82.70% <ø> (ø)
mapreduce 81.35% <ø> (ø)
marathon 83.43% <ø> (ø)
mcache 93.50% <ø> (ø)
mesos_master 89.75% <ø> (ø)
mesos_slave 93.63% <ø> (ø)
mongo 95.94% <ø> (ø)
nagios 89.01% <ø> (ø)
network 93.63% <ø> (+1.08%) ⬆️
nfsstat 95.20% <ø> (ø)
nginx 95.24% <ø> (+0.54%) ⬆️
nginx_ingress_controller 98.36% <ø> (ø)
openldap 96.79% <ø> (ø)
openmetrics 98.08% <ø> (ø)
openstack 51.45% <ø> (ø)
openstack_controller 91.12% <ø> (ø)
pgbouncer 91.33% <ø> (ø)
php_fpm 90.25% <ø> (+0.84%) ⬆️
postfix 88.04% <ø> (ø)
powerdns_recursor 96.65% <ø> (ø)
presto 59.09% <ø> (ø)
process 85.42% <ø> (+0.28%) ⬆️
prometheus 94.17% <ø> (ø)
proxysql 98.97% <ø> (ø)
pulsar 100.00% <ø> (ø)
redisdb 87.78% <ø> (ø)
rethinkdb 97.93% <ø> (ø)
riak 99.22% <ø> (ø)
riakcs 93.61% <ø> (ø)
silk 93.82% <ø> (ø)
singlestore 90.81% <ø> (ø)
snowflake 96.61% <ø> (ø)
solr 56.25% <ø> (ø)
spark 93.91% <ø> (ø)
squid 100.00% <ø> (ø)
ssh_check 91.58% <ø> (ø)
statsd 87.36% <ø> (+1.05%) ⬆️
strimzi 89.70% <ø> (ø)
supervisord 90.14% <ø> (ø)
system_core 92.66% <ø> (ø)
system_swap 98.30% <ø> (ø)
tcp_check 92.92% <ø> (+1.34%) ⬆️
teamcity 88.74% <ø> (+3.21%) ⬆️
temporal 100.00% <ø> (ø)
teradata 94.06% <ø> (ø)
tls 92.18% <ø> (+0.82%) ⬆️
tokumx 58.40% <ø> (ø)
tomcat 61.29% <ø> (ø)
torchserve 97.33% <ø> (ø)
traffic_server 96.13% <ø> (ø)
twemproxy 79.45% <ø> (ø)
twistlock 79.62% <ø> (ø)
varnish 84.39% <ø> (+0.26%) ⬆️
vault 95.53% <ø> (+0.57%) ⬆️
vertica 98.34% <ø> (ø)
vsphere 95.81% <ø> (+0.06%) ⬆️
weaviate 76.27% <ø> (ø)
weblogic 71.73% <ø> (ø)
win32_event_log 42.99% <ø> (+0.55%) ⬆️
windows_performance_counters 98.36% <ø> (ø)
windows_service 98.00% <ø> (ø)
wmi_check 92.91% <ø> (ø)
yarn 89.59% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

def _hash_to_hex(hash):
return to_native_string(binascii.hexlify(hash))
def _hash_to_hex(hash) -> str:
return binascii.hexlify(hash).decode("utf-8")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to_native_string converts bytes b'xxx' to "b'xxx'". The correct str representation of the hash should be properly decoded.

@lu-zhengda lu-zhengda marked this pull request as ready for review September 5, 2023 18:33
@lu-zhengda lu-zhengda requested review from a team as code owners September 5, 2023 18:33
@lu-zhengda lu-zhengda self-assigned this Sep 5, 2023
estherk15
estherk15 previously approved these changes Sep 5, 2023
@github-actions
Copy link

github-actions bot commented Sep 5, 2023

Test Results

  1 009 files    1 009 suites   8h 34m 1s ⏱️
  5 962 tests   5 894 ✔️      68 💤 0
25 387 runs  21 030 ✔️ 4 357 💤 0

Results for commit eb058ff.

♻️ This comment has been updated with latest results.

jmeunier28
jmeunier28 previously approved these changes Sep 5, 2023
Copy link
Member

@FlorentClarret FlorentClarret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we had unit tests to confirm the bug is fixed?

FlorentClarret
FlorentClarret previously approved these changes Sep 8, 2023
Copy link
Member

@FlorentClarret FlorentClarret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for adding the tests!

I left one suggestion, but feel free to merge as it is if you want

datadog_checks_base/tests/base/utils/db/test_util.py Outdated Show resolved Hide resolved
@lu-zhengda lu-zhengda merged commit 3009b18 into master Sep 8, 2023
336 of 337 checks passed
@lu-zhengda lu-zhengda deleted the zhengda.lu/activity-payload-bytes branch September 8, 2023 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants