HTTP plugin code 400 on connection to Citrix ADC #11478

SpiderD555 · 2022-07-08T08:35:36Z

Relevant telegraf.conf

[[inputs.http]]
  urls = [
        "https://netscaler/nitro/v1/stat/lbvserver"
        ]
  method = "GET"

  cookie_auth_url = "https://netscaler/nitro/v1/config/login"
  cookie_auth_method = "POST"
  cookie_auth_headers = { Content-Type = "application/json"}
  cookie_auth_body = '{"login":{"username": "username", "password":"password"}}'
  ## cookie_auth_renewal not set or set to "0" will auth once and never renew the cookie
  cookie_auth_renewal = "1h"
  data_format = "json_v2"

  insecure_skip_verify = true

Logs from Telegraf

E! [telegraf] Error running agent: could not initialize input inputs.http: cookie auth renewal received status code: 400 (Bad Request)

System info

Telegraf 1.22.4 (also tried 1.24.0~e945364b), Redhat 8.6

Docker

No response

Steps to reproduce

Configure HTTP plugin as shown in example
Configure Citrix ADC with basic credentials
Start Telegraf
Telegraf stops with error

Expected behavior

I expect Telegraf HTTP plugin will connect to Citrix ADC and initiate the session, so I can draw stats using API calls

Moreover Telegraf stops on that error code 400, which in my humble opinion may be a bad design choice since it may crash otherwise a perfectly fine running Telegraf instance. Telegraf should just ignore the error and maybe retry later instead of stopping entirely. Imagine a situation where we have 1 HTTP endpoint and 99 SNMP devices. When we poll a device with HTTP plugin and it replies with non-200 code (because of some internal error), then whole Telegraf instance crashes and stops gathering statistics for those 99 perfectly working devices.

Actual behavior

E! [telegraf] Error running agent: could not initialize input inputs.http: cookie auth renewal received status code: 400 (Bad Request)

Additional info

When I use curl then it all works:

Establishing new session
curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'
HTTP/1.1 201 Created
Date: Fri, 08 Jul 2022 06:56:16 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%23465611085B10AEECBC084499AA564FABA40BACAE435A343F2E40A96A12191918377A434FE2959F11C56076F0E08DDA840EFF0A9968DA9BAA0CA70555A6D168D5155CEBBAB02F9101FBED406B01164D0E617790FE785EF572F90C11FBDFE84FA7437466B824ECA2BC9652FB39FFCAF3DEF23B3AD421A53C32A8D52F75B874; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##465611085B10AEECBC084499AA564FABA40BACAE435A343F2E40A96A12191918377A434FE2959F11C56076F0E08DDA840EFF0A9968DA9BAA0CA70555A6D168D5155CEBBAB02F9101FBED406B01164D0E617790FE785EF572F90C11FBDFE84FA7437466B824ECA2BC9652FB39FFCAF3DEF23B3AD421A53C32A8D52F75B874" }

Getting example statistics
curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%23465611085B10AEECBC084499AA564FABA40BACAE435A343F2E40A96A12191918377A434FE2959F11C56076F0E08DDA840EFF0A9968DA9BAA0CA70555A6D168D5155CEBBAB02F9101FBED406B01164D0E617790FE785EF572F90C11FBDFE84FA7437466B824ECA2BC9652FB39FFCAF3DEF23B3AD421A53C32A8D52F75B874; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Fri, 08 Jul 2022 06:57:05 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

At first I thought that maybe Telegraf just won't accept code 201 as per #11134
so I tested new build #11472
but it still fails

The text was updated successfully, but these errors were encountered:

fixes: influxdata#11478

powersj · 2022-07-08T14:51:02Z

Hi,

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'
HTTP/1.1 201 Created

It does look like the system you are connecting to returns a 201, which will eventually require the fix in #11472. However, Telegraf is not getting that far and instead getting a 400 per your telegraf log. The Citrix ADC docs say the reason is in the body, so let's try to find out why.

I have put up #11482, which should have some artifacts attached shortly. Can you give that a try and see what the body says?

SpiderD555 · 2022-07-11T07:12:23Z

Hi,

Same story with latest build (with more debug information this time):

Jul 11 08:59:54 localhost telegraf[2894337]: 2022-07-11T06:59:54Z E! [telegraf] Error running agent: could not initialize input inputs.http: cookie auth renewal received status code: 400 (Bad Request) - <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Jul 11 08:59:54 localhost telegraf[2894337]: <html><head>
Jul 11 08:59:54 localhost telegraf[2894337]: <title>400 Bad Request</title>
Jul 11 08:59:54 localhost telegraf[2894337]: </head><body>
Jul 11 08:59:54 localhost telegraf[2894337]: <h1>Bad Request</h1>
Jul 11 08:59:54 localhost telegraf[2894337]: <p>Your browser sent a request that this server could not understand.<br />
Jul 11 08:59:54 localhost telegraf[2894337]: </p>
Jul 11 08:59:54 localhost telegraf[2894337]: </body></html>
Jul 11 08:59:54 localhost telegraf[2894337]: {"errorcode":-1,"message":"Invalid payload format. Default format is json","severity":"ERROR"}

Because of "Invalid payload format. Default format is json" I think we may need to see exactly what Telegraf is sending to Citrix ADC, to get to the root cause of the problem.

powersj · 2022-07-11T15:26:58Z

Thanks for trying that out!

We saw something similar in #11083 and learned that some devices are expected the content length and did not like a request formed by Go's NopCloser. Given this is for a request to generate a cookie, the request will not be huge, and there is no need to use the NopCloser.

I have pushed an update to the PR that will change the body to use a io.Reader over an io.ReadCloser and also print out some debug information before doing the request.

Can you give that update a shot once new artifacts are attached?

SpiderD555 · 2022-07-12T10:20:12Z

I am not sure which artifact you are referring to. I tried this one: #11482

Anyway the result is following: Telegraf is not stopping, but I also don't see any data from the Citrix device in my InfluxDB instance (however I do get data from few test SNMP probes).
The only log I see is only this:

https://netscaler/nitro/v1/config/login
POST
map[Content-Type:[application/json]]

So far I didn't see any other connection attempts to Citrix in logs, so I assume that it authenticates successfully, but it is not sending any proper API calls to draw data.

Now the situation is getting a bit more interesting when I set cookie_auth_renewal = "5m"

I am getting inital session authentication, then Telegraf tries to renew the session with old session cookie (which I suppose it shouldn't as per the log):


Jul 12 12:03:27 localhost telegraf[2974996]: 2022-07-12T10:03:27Z D! [agent] Initializing plugins
Jul 12 12:03:27 localhost telegraf[2974996]: https://netscaler/nitro/v1/config/login
Jul 12 12:03:27 localhost telegraf[2974996]: POST
Jul 12 12:03:27 localhost telegraf[2974996]: map[Content-Type:[application/json]]

Jul 12 12:08:27 localhost telegraf[2974996]: https://netscaler/nitro/v1/config/login
Jul 12 12:08:27 localhost telegraf[2974996]: POST
Jul 12 12:08:27 localhost telegraf[2974996]: map[Content-Type:[application/json] Cookie:[sessionid=%23%238543669E93EC9727F77E001DCF46FF973974DCBD31F174C64A6DC
F2302E28B28E86C6B5B9867FD642CA6C2856D138218E78151433B843E4DF51D4BE87C5662B9B79495237AE13ECD4B6D8CF12553552D37AAD71F59BF3312878E8FB88E650928DADA7322F5DFE4BACC88713E0F5A29ADA1B52DD7D8386
E61F5FD2FE3A41D]]
Jul 12 12:08:27 localhost telegraf[2974996]: 2022-07-12T10:08:27Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/c
onfig/login": cookie auth renewal received status code: 401 (Unauthorized) - { "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }

Jul 12 12:13:27 localhost telegraf[2974996]: https://netscaler/nitro/v1/config/login
Jul 12 12:13:27 localhost telegraf[2974996]: POST
Jul 12 12:13:27 localhost telegraf[2974996]: map[Content-Type:[application/json] Cookie:[sessionid=%23%238543669E93EC9727F77E001DCF46FF973974DCBD31F174C64A6DC
F2302E28B28E86C6B5B9867FD642CA6C2856D138218E78151433B843E4DF51D4BE87C5662B9B79495237AE13ECD4B6D8CF12553552D37AAD71F59BF3312878E8FB88E650928DADA7322F5DFE4BACC88713E0F5A29ADA1B52DD7D8386
E61F5FD2FE3A41D]]
Jul 12 12:13:27 localhost telegraf[2974996]: 2022-07-12T10:13:27Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/c
onfig/login": cookie auth renewal received status code: 401 (Unauthorized) - { "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }

powersj · 2022-07-12T14:59:02Z

So far I didn't see any other connection attempts to Citrix in logs, so I assume that it authenticates successfully, but it is not sending any proper API calls to draw data.

It sounds like progress and indeed sounds like the Cirtix server was possibly expecting the content length similar to #11083.

I am getting inital session authentication, then Telegraf tries to renew the session with old session cookie (which I suppose it shouldn't as per the log):

Can you demonstrate a renewal with curl 1) with the cookie in the headers and 2) without the cookie in the headers, please? The 401 makes it pretty clear that the user credentials are wrong on renewal, so it seems unable to get a new cookie which is why the old cookie is used.

Similar to influxdata#11083, some HTTP servers seem to quietly requires the content-length header in messages sent. Use of Go's ReadCloser does not allow for the content-length to be sent. As this is for a simple login the body length will be very small and the more efficient processing of a closer is not required in this use case. This also adds the ability to see the response body in the event of an error to gain a bit more details. fixes: influxdata#11478

SpiderD555 · 2022-07-13T06:41:42Z

Let me demonstrate 2)

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'
HTTP/1.1 201 Created
Date: Wed, 13 Jul 2022 06:07:51 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%238F90C1B1BEE9369035623B2BFD6448F32C7F2EC3FDCFF3D81C5847DB174ADEA8EA277F4D6AC7695E37C829C5E0BA0675E81BD15DCC4C226CE3A4EAD38F64B40E4CD5C2623775D0A4070EEDA0434DB0657BAD7AED4933D57AC75B6790EF946F0F1FBE1A1D0C35F7E406D1C5FADD5242585B6163DC8E039EE19166F27F68A3; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##8F90C1B1BEE9369035623B2BFD6448F32C7F2EC3FDCFF3D81C5847DB174ADEA8EA277F4D6AC7695E37C829C5E0BA0675E81BD15DCC4C226CE3A4EAD38F64B40E4CD5C2623775D0A4070EEDA0434DB0657BAD7AED4933D57AC75B6790EF946F0F1FBE1A1D0C35F7E406D1C5FADD5242585B6163DC8E039EE19166F27F68A3" }

=============================================

curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%238F90C1B1BEE9369035623B2BFD6448F32C7F2EC3FDCFF3D81C5847DB174ADEA8EA277F4D6AC7695E37C829C5E0BA0675E81BD15DCC4C226CE3A4EAD38F64B40E4CD5C2623775D0A4070EEDA0434DB0657BAD7AED4933D57AC75B6790EF946F0F1FBE1A1D0C35F7E406D1C5FADD5242585B6163DC8E039EE19166F27F68A3; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 06:08:55 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

=============================================

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'
HTTP/1.1 201 Created
Date: Wed, 13 Jul 2022 06:09:37 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%2368BD0AF9A3A9134600B0C207E7EBED684541F64FF85884273CF676CB9D411DF693B4F1420EB77BE061F1D655C2486E025A05E26D14A513D1FBD6641D218F42EA9B8FB05619A2AD2715DD0A6AC95CCA11D44BD7B1A082095F84629C295542431A4DA9BC902A5762E60849037EEDD2EE4C9C0D05B8D600B9040822D6BEA804; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##68BD0AF9A3A9134600B0C207E7EBED684541F64FF85884273CF676CB9D411DF693B4F1420EB77BE061F1D655C2486E025A05E26D14A513D1FBD6641D218F42EA9B8FB05619A2AD2715DD0A6AC95CCA11D44BD7B1A082095F84629C295542431A4DA9BC902A5762E60849037EEDD2EE4C9C0D05B8D600B9040822D6BEA804" }

=============================================

curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%2368BD0AF9A3A9134600B0C207E7EBED684541F64FF85884273CF676CB9D411DF693B4F1420EB77BE061F1D655C2486E025A05E26D14A513D1FBD6641D218F42EA9B8FB05619A2AD2715DD0A6AC95CCA11D44BD7B1A082095F84629C295542431A4DA9BC902A5762E60849037EEDD2EE4C9C0D05B8D600B9040822D6BEA804; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 06:10:41 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

I think there is no such process to go with 1) method. Citrix ADC API documentation says you can logout/terminate a session using a cookie, but to initiate new session it is required to send login and password without previous session cookie. Old session will just time-out on Citrix ADC at some point of time. In the example above Curl does use a cookie to initiate a session SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/ but according to my knowledge setting cookie expiration date in the past is like saying it is expired/not for use.
Therefore each new session should be initiated from scratch in my humble opinion.

powersj · 2022-07-13T14:31:18Z

Thanks for the response. I've pushed another update just now that will omit the cookie from the headers during an auth. Once new artifacts build, can you give that one more shot?

Thanks again!

SpiderD555 · 2022-07-14T07:34:14Z

Similar story with the latest artifact, although there is no mention about cookie in the reply from Citrix ADC this time.

Jul 14 08:42:13 localhost telegraf[3105480]: 2022-07-14T06:42:13Z I! Loaded inputs: http snmp (15x)
Jul 14 08:47:15 localhost telegraf[3105480]: 2022-07-14T06:47:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 08:52:15 localhost telegraf[3105480]: 2022-07-14T06:52:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 08:57:15 localhost telegraf[3105480]: 2022-07-14T06:57:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 09:02:15 localhost telegraf[3105480]: 2022-07-14T07:02:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]
Jul 14 09:07:15 localhost telegraf[3105480]: 2022-07-14T07:07:15Z E! [inputs.http] renewal failed for "https://netscaler/nitro/v1/config/login": cookie auth renewal received status code: 401 (Unauthorized) [{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }]

powersj · 2022-07-14T13:31:37Z

Hmm I re-added the debug info to print out the headers. Maybe it is getting set again? Can you try one more time? I am not sure what else this might be.

Your CURL examples in your previous comment omitted the example of logging in with the cookie in the header. I wanted to see if you could reproduce the same failure on the CLI.

SpiderD555 · 2022-07-14T15:31:26Z

Below is the flow you request. First is login, then regular API call, third request is a second authentication attempt.

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}'

HTTP/1.1 201 Created
Date: Thu, 14 Jul 2022 15:22:27 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: SESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: sessionid=%23%238DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7; path=/nitro/v1
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 328
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "sessionid": "##8DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7" }

=============================================

curl -ik https://netscaler/nitro/v1/stat/lbvserver -H 'Content-Type: application/json' --cookie "sessionid=%23%238DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7; path=/nitro/v1"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 06:08:55 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1640
Content-Type: application/json; charset=utf-8

{ "errorcode": 0, "message": "Done", "severity": "NONE", "lbvserver": [ { "name": "lb-test-ssl", "avgcltttlb": "0", "cltresponsetimeapdex": 1.000000, "vsvrsurgecount": "0", "establishedconn": "0", "inactsvcs": "0", "vslbhealth": "0", "primaryipaddress": "169.254.1.1", "primaryport": 443, "type": "SSL", "state": "DOWN", "actsvcs": "0", "cpuusagepm": "0", "tothits": "0", "hitsrate": 0, "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "totalh2requests": "0", "h2requestsrate": 0, "totalh2responses": "0", "h2responsesrate": 0, "totalpktsrecvd": "0", "pktsrecvdrate": 0, "totalpktssent": "0", "pktssentrate": 0, "curclntconnections": "0", "cursrvrconnections": "0", "curpersistencesessions": "0", "curbackuppersistencesessions": "0", "surgecount": "0", "svcsurgecount": "0", "sothreshold": "0", "totspillovers": "0", "labelledconn": "0", "pushlabel": "0", "deferredreq": "0", "deferredreqrate": 0, "invalidrequestresponse": "0", "invalidrequestresponsedropped": "0", "totvserverdownbackuphits": "0", "curmptcpsessions": "0", "cursubflowconn": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "totalsvrbusyerr": "0", "svrbusyerrrate": 0, "reqretrycount": "0", "reqretrycountexceeded": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "totcltttlbtransactions": "0", "cltttlbtransactionsrate": 0, "toleratingttlbtransactions": "0", "toleratingttlbtransactionsrate": 0, "frustratingttlbtransactions": "0", "frustratingttlbtransactionsrate": 0 } ] }

=============================================

curl -ik https://netscaler/nitro/v1/config/login -X POST -H 'Content-Type: application/json' -d '{ "login" :{"username": "username", "password": "password"}}' --cookie "sessionid=%23%238DDDC7D90D1BB7A5973D89EDB1DEB52A545D7D4EBB19684B16C5C2375342D9681DD03752A058F240120DC1DC6183256E12992765BB50515DBD32E874F6A587A5413AB425317064910B4678CCC088F16944A3FDA1F9553F89DCA33431FDBF49856692929B04140A8F3E0386988F56E3472471D061D1F78321F43FEC68EDB7; path=/nitro/v1"
HTTP/1.1 401 Unauthorized
Date: Thu, 14 Jul 2022 15:24:07 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Feature-Policy: camera 'none'; microphone 'none'; geolocation 'none'
Referrer-Policy: no-referrer
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 84
Content-Type: application/json; charset=utf-8

{ "errorcode": 354, "message": "Invalid username or password", "severity": "ERROR" }

powersj · 2022-07-14T15:51:45Z

thank you! That does show that setting the cookie is causing a 401.

I just pushed another change which will recreate the cookie jar at every reauth. This should ensure the cookie is never used during the auth process.

SpiderD555 · 2022-07-15T09:38:14Z

It looks better now. Session renewal is working. I see that because of those messages:

Jul 15 11:16:15 localhost telegraf[3180145]: map[Content-Type:[application/json]]
Jul 15 11:21:15 localhost telegraf[3180145]: map[Content-Type:[application/json]]
Jul 15 11:26:15 localhost telegraf[3180145]: map[Content-Type:[application/json]]

However I still don't see any real statistic data from Citrix ADC.

Similar to influxdata#11083, some HTTP servers seem to quietly requires the content-length header in messages sent. Use of Go's ReadCloser does not allow for the content-length to be sent. As this is for a simple login the body length will be very small and the more efficient processing of a closer is not required in this use case. Moves the creation of the cookie jar as well. On a re-auth, the cookie was included in the request. Not only does this not make sense it can cause issues on some APIs where this was not expected. Given the original request does not contain this it seems safe to remake the cookie jar, emptying it, for each auth request. This also adds the ability to see the response body in the event of an error to gain a bit more details. fixes: influxdata#11478

powersj · 2022-07-15T13:28:48Z

It looks better now. Session renewal is working.

Awesome! Thanks for hanging in there. I am going to get this branch merged since it already fixes a couple different issues.

However I still don't see any real statistic data from Citrix ADC.

Based on your previous messages you are getting JSON back that looks like:

{
  "errorcode": 0,
  "message": "Done",
  "severity": "NONE",
  "lbvserver": [
    {
      "name": "lb-test-ssl",
      "avgcltttlb": "0",
      "cltresponsetimeapdex": 1,
      "vsvrsurgecount": "0",
      "establishedconn": "0",
      "inactsvcs": "0",
      "vslbhealth": "0",
      "primaryipaddress": "169.254.1.1",
      "primaryport": 443,
      "type": "SSL",
      "state": "DOWN",
      "actsvcs": "0",
      "cpuusagepm": "0",
      "tothits": "0",
      "hitsrate": 0,
      "totalrequests": "0",
      "requestsrate": 0,
      "totalresponses": "0",
      "responsesrate": 0,
      "totalrequestbytes": "0",
      "requestbytesrate": 0,
      "totalresponsebytes": "0",
      "responsebytesrate": 0,
      "totalh2requests": "0",
      "h2requestsrate": 0,
      "totalh2responses": "0",
      "h2responsesrate": 0,
      "totalpktsrecvd": "0",
      "pktsrecvdrate": 0,
      "totalpktssent": "0",
      "pktssentrate": 0,
      "curclntconnections": "0",
      "cursrvrconnections": "0",
      "curpersistencesessions": "0",
      "curbackuppersistencesessions": "0",
      "surgecount": "0",
      "svcsurgecount": "0",
      "sothreshold": "0",
      "totspillovers": "0",
      "labelledconn": "0",
      "pushlabel": "0",
      "deferredreq": "0",
      "deferredreqrate": 0,
      "invalidrequestresponse": "0",
      "invalidrequestresponsedropped": "0",
      "totvserverdownbackuphits": "0",
      "curmptcpsessions": "0",
      "cursubflowconn": "0",
      "totalconnreassemblyqueue75": "0",
      "totalconnreassemblyqueueflush": "0",
      "totalsvrbusyerr": "0",
      "svrbusyerrrate": 0,
      "reqretrycount": "0",
      "reqretrycountexceeded": "0",
      "httpmaxhdrszpkts": "0",
      "httpmaxhdrfldlenpkts": "0",
      "tcpmaxooopkts": "0",
      "totcltttlbtransactions": "0",
      "cltttlbtransactionsrate": 0,
      "toleratingttlbtransactions": "0",
      "toleratingttlbtransactionsrate": 0,
      "frustratingttlbtransactions": "0",
      "frustratingttlbtransactionsrate": 0
    }
  ]
}

As you are using the json_v2 parser, you are going to need to specify some settings to correctly parse some of the data provided to you:

  data_format = "json_v2"
  [[inputs.http.json_v2]]
    [[inputs.http.json_v2.object]]
      path = "lbvserver"

Which should produce something similar to this below. Note I used the file input to produce this example:

file name="lb-test-ssl",avgcltttlb="0",cltresponsetimeapdex=1,vsvrsurgecount="0",establishedconn="0",inactsvcs="0",vslbhealth="0",primaryipaddress="169.254.1.1",primaryport=443,type="SSL",state="DOWN",actsvcs="0",cpuusagepm="0",tothits="0",hitsrate=0,totalrequests="0",requestsrate=0,totalresponses="0",responsesrate=0,totalrequestbytes="0",requestbytesrate=0,totalresponsebytes="0",responsebytesrate=0,totalh2requests="0",h2requestsrate=0,totalh2responses="0",h2responsesrate=0,totalpktsrecvd="0",pktsrecvdrate=0,totalpktssent="0",pktssentrate=0,curclntconnections="0",cursrvrconnections="0",curpersistencesessions="0",curbackuppersistencesessions="0",surgecount="0",svcsurgecount="0",sothreshold="0",totspillovers="0",labelledconn="0",pushlabel="0",deferredreq="0",deferredreqrate=0,invalidrequestresponse="0",invalidrequestresponsedropped="0",totvserverdownbackuphits="0",curmptcpsessions="0",cursubflowconn="0",totalconnreassemblyqueue75="0",totalconnreassemblyqueueflush="0",totalsvrbusyerr="0",svrbusyerrrate=0,reqretrycount="0",reqretrycountexceeded="0",httpmaxhdrszpkts="0",httpmaxhdrfldlenpkts="0",tcpmaxooopkts="0",totcltttlbtransactions="0",cltttlbtransactionsrate=0,toleratingttlbtransactions="0",toleratingttlbtransactionsrate=0,frustratingttlbtransactions="0",frustratingttlbtransactionsrate=0 1657891626000000000

From there you can continue to customize these settings further.

SpiderD555 · 2022-07-15T14:23:45Z

I can confirm it is working. I finally get data in InfluxDB from Citrix ADC. Thank you for your help

SpiderD555 added the bug unexpected problem or unintended behavior label Jul 8, 2022

powersj added a commit to powersj/telegraf that referenced this issue Jul 8, 2022

fix: get cookie response body on error

49c19f3

fixes: influxdata#11478

powersj mentioned this issue Jul 8, 2022

fix: use reader over readcloser, regen cookie-jar #11482

Merged

powersj added the waiting for response waiting for response from contributor label Jul 8, 2022

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 11, 2022

powersj added the waiting for response waiting for response from contributor label Jul 11, 2022

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 12, 2022

powersj added the waiting for response waiting for response from contributor label Jul 12, 2022

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 13, 2022

powersj added the waiting for response waiting for response from contributor label Jul 13, 2022

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 14, 2022

powersj added the waiting for response waiting for response from contributor label Jul 14, 2022

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 14, 2022

powersj added the waiting for response waiting for response from contributor label Jul 14, 2022

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 15, 2022

SpiderD555 closed this as completed Jul 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP plugin code 400 on connection to Citrix ADC #11478

HTTP plugin code 400 on connection to Citrix ADC #11478

SpiderD555 commented Jul 8, 2022

powersj commented Jul 8, 2022

SpiderD555 commented Jul 11, 2022 •

edited

Loading

powersj commented Jul 11, 2022

SpiderD555 commented Jul 12, 2022 •

edited

Loading

powersj commented Jul 12, 2022

SpiderD555 commented Jul 13, 2022

powersj commented Jul 13, 2022

SpiderD555 commented Jul 14, 2022

powersj commented Jul 14, 2022

SpiderD555 commented Jul 14, 2022

powersj commented Jul 14, 2022

SpiderD555 commented Jul 15, 2022

powersj commented Jul 15, 2022

SpiderD555 commented Jul 15, 2022

HTTP plugin code 400 on connection to Citrix ADC #11478

HTTP plugin code 400 on connection to Citrix ADC #11478

Comments

SpiderD555 commented Jul 8, 2022

Relevant telegraf.conf

Logs from Telegraf

System info

Docker

Steps to reproduce

Expected behavior

Actual behavior

Additional info

powersj commented Jul 8, 2022

SpiderD555 commented Jul 11, 2022 • edited Loading

powersj commented Jul 11, 2022

SpiderD555 commented Jul 12, 2022 • edited Loading

powersj commented Jul 12, 2022

SpiderD555 commented Jul 13, 2022

powersj commented Jul 13, 2022

SpiderD555 commented Jul 14, 2022

powersj commented Jul 14, 2022

SpiderD555 commented Jul 14, 2022

powersj commented Jul 14, 2022

SpiderD555 commented Jul 15, 2022

powersj commented Jul 15, 2022

SpiderD555 commented Jul 15, 2022

SpiderD555 commented Jul 11, 2022 •

edited

Loading

SpiderD555 commented Jul 12, 2022 •

edited

Loading