Upgrade pymqi to 1.10.1 (#5955)

* Add api version config option * Use api version on integration tests * Rename api_version to mqcd_version * fix config file * Fix style * Rename api version in tests * Fix unit tests * Bump pymq * Improve documentation, add gcc to pipelines * Install gcc on agent image when running e2e * Add C include path to E2E Use new IBM MQ client lib Update IBM MQ readme Move all tarbals to azure * Bump pymqi * Update common.py
DataDog · Mar 3, 2020 · e9accaf · e9accaf
1 parent 74084f3
commit e9accaf
Show file tree

Hide file tree

Showing 6 changed files with 22 additions and 11 deletions.
diff --git a/.azure-pipelines/scripts/ibm_mq/linux/55_install_client.sh b/.azure-pipelines/scripts/ibm_mq/linux/55_install_client.sh
@@ -1,9 +1,13 @@
 #!/bin/bash
 
+# This script installs IBM MQ development version on the CI machines to be able to
+# * Compile pymqi image
+# * Run integration tests on the machine
+
 set -ex
 
 TMP_DIR=/tmp/mq
-MQ_URL=https://s3.amazonaws.com/dd-agent-tarball-mirror/mqadv_dev90_linux_x86-64.tar.gz
+MQ_URL=https://ddintegrations.blob.core.windows.net/ibm-mq/mqadv_dev90_linux_x86-64.tar.gz
 MQ_PACKAGES="MQSeriesRuntime-*.rpm MQSeriesServer-*.rpm MQSeriesMsg*.rpm MQSeriesJava*.rpm MQSeriesJRE*.rpm MQSeriesGSKit*.rpm"
 
 if [ -e /opt/mqm/inc/cmqc.h ]; then
@@ -21,6 +25,7 @@ sudo apt-get install -y --no-install-recommends \
   debianutils \
   findutils \
   gawk \
+  gcc \
   grep \
   libc-bin \
   mount \

diff --git a/datadog_checks_base/datadog_checks/base/data/agent_requirements.in b/datadog_checks_base/datadog_checks/base/data/agent_requirements.in
@@ -41,7 +41,7 @@ pyasn1==0.4.6
 pycryptodomex==3.9.4
 pyhdb==0.3.4
 pymongo==3.8.0
-pymqi==1.8.0; sys_platform != 'win32' and sys_platform != 'darwin'
+pymqi==1.10.1; sys_platform != 'win32' and sys_platform != 'darwin'
 pymysql==0.9.3
 pyodbc==4.0.26; sys_platform != 'darwin'
 pyro4==4.73; sys_platform == 'win32'

diff --git a/ibm_mq/README.md b/ibm_mq/README.md
@@ -12,13 +12,15 @@ The IBM MQ check is included in the [Datadog Agent][2] package.
 
 To use the IBM MQ check, you need to:
 
-1. Make sure the [IBM MQ Client][3] is installed (unless the IBM MQ server is already installed).
-2. Update your LD_LIBRARY_PATH to include the location of the libraries
+1. Make sure the [IBM MQ Client][3] 9.1+ is installed (unless the IBM MQ server is already installed).
+2. Update your LD_LIBRARY_PATH and C_INCLUDE_PATH to include the location of the libraries
 
 For example:
 
 ```text
 export LD_LIBRARY_PATH=/opt/mqm/lib64:/opt/mqm/lib:$LD_LIBRARY_PATH
+export C_INCLUDE_PATH=/opt/mqm/inc
+
 ```
 
 **Note**: Agent v6+ uses Upstart or systemd to orchestrate the datadog-agent service. Environment variables may need to be added to the service configuration files at the default locations of:

diff --git a/ibm_mq/requirements.in b/ibm_mq/requirements.in
@@ -1 +1 @@
-pymqi==1.8.0; sys_platform != 'win32' and sys_platform != 'darwin'
+pymqi==1.10.1; sys_platform != 'win32' and sys_platform != 'darwin'
diff --git a/ibm_mq/tests/README.md b/ibm_mq/tests/README.md
@@ -3,14 +3,15 @@ How to install pymqi in MacOs
 To install pymqi on Mac, first set up the [IBM MQ toolkit for Mac OS][1].
 
 1. Download and extract the [toolkit][2].
-2. Set up the following environment variables:
+2. Create a softlink from MQ_INSTALLATION_PATH to /opt/mqm
+3. Set up the following environment variables:
     ```
     export MQ_INSTALLATION_PATH=~/IBM-MQ-Toolkit-Mac-x64-9.1.2.0
     export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:$MQ_INSTALLATION_PATH/lib64
+    export C_INCLUDE_PATH=/opt/mqm/inc
     export PATH="${PATH}:$MQ_INSTALLATION_PATH/bin"
     ```
-3. Create a softlink from MQ_INSTALLATION_PATH to /opt/mqm
-4. `pip install pymqi`
+4. `pip install pymqi==1.10.1`
 5. Edit `requirements.in` and remove `and sys_platform != 'darwin'` so `pymqi` gets autoinstalled on tox environments.
 Do not commit this change as it would break the mac agent build.
 

diff --git a/ibm_mq/tests/common.py b/ibm_mq/tests/common.py
@@ -85,13 +85,16 @@
 }
 
 E2E_METADATA = {
+    # This script makes the necessary setup to be able to compile pymqi on the agent machine
     'start_commands': [
+        'apt-get update',
+        'apt-get install gcc -y',
         'mkdir /opt/mqm',
-        'curl -o /opt/mqm/mq-client.tar.gz '
-        'https://dd-agent-tarball-mirror.s3.amazonaws.com/9.0.0.6-IBM-MQC-Redist-LinuxX64.tar.gz',
+        'curl -L -o /opt/mqm/mq-client.tar.gz '
+        'https://ddintegrations.blob.core.windows.net/ibm-mq/9.1.0.4-IBM-MQC-Redist-LinuxX64.tar.gz',
         'tar -C /opt/mqm -xf /opt/mqm/mq-client.tar.gz',
     ],
-    'env_vars': {'LD_LIBRARY_PATH': '/opt/mqm/lib64:/opt/mqm/lib'},
+    'env_vars': {'LD_LIBRARY_PATH': '/opt/mqm/lib64:/opt/mqm/lib', 'C_INCLUDE_PATH': '/opt/mqm/inc'},
 }
 
 QUEUE_METRICS = [