Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpUtility::SendJsonBody(): auto-sanitize non-UTF-8 JSON #6809

Closed
wants to merge 1 commit into from

Conversation

Al2Klimov
Copy link
Member

fixes #4703

@Al2Klimov Al2Klimov requested a review from dnsmichi November 29, 2018 17:37
@Al2Klimov
Copy link
Member Author

Test protocol

  1. Build Icinga 2
  2. Setup API
  3. Start the daemon
  4. Create a host via API
  5. Stop the daemon

Both the API response and the state file look fine.

@Al2Klimov Al2Klimov removed the request for review from dnsmichi November 30, 2018 09:00
@Al2Klimov Al2Klimov changed the title HttpUtility::SendJsonBody(): auto-sanitize non-UTF-8 JSON WIP: HttpUtility::SendJsonBody(): auto-sanitize non-UTF-8 JSON Nov 30, 2018
@Al2Klimov Al2Klimov force-pushed the bugfix/non-utf-8-broken-json-4703 branch from 47b7d99 to 154df7b Compare November 30, 2018 09:39
@Al2Klimov Al2Klimov changed the title WIP: HttpUtility::SendJsonBody(): auto-sanitize non-UTF-8 JSON HttpUtility::SendJsonBody(): auto-sanitize non-UTF-8 JSON Nov 30, 2018
@Al2Klimov Al2Klimov requested a review from dnsmichi November 30, 2018 09:39
@Al2Klimov
Copy link
Member Author

In addition the API query results differ.

$ hexdump -C <lolcat.conf
00000000  6f 62 6a 65 63 74 20 48  6f 73 74 20 22 6c 6f 6c  |object Host "lol|
00000010  63 61 74 22 20 7b 0a 09  63 68 65 63 6b 5f 63 6f  |cat" {..check_co|
00000020  6d 6d 61 6e 64 20 3d 20  22 68 6f 73 74 61 6c 69  |mmand = "hostali|
00000030  76 65 22 0a 09 76 65 72  73 69 6f 6e 20 3d 20 31  |ve"..version = 1|
00000040  35 34 33 35 37 30 36 35  37 2e 33 32 30 33 30 36  |543570657.320306|
00000050  0a 09 7a 6f 6e 65 20 3d  20 22 43 45 4e 53 4f 52  |..zone = "CENSOR|
00000060  45 44 22 0a 09 76 61 72  73 2e 66 6f 6f 62 61 72  |ED"..vars.foobar|
00000070  20 3d 20 22 80 81 82 83  84 85 86 87 88 89 8a 8b  | = "............|
00000080  8c 8d 8e 8f 90 91 92 93  94 95 96 97 98 99 9a 9b  |................|
00000090  9c 9d 9e 9f a0 a1 a2 a3  a4 a5 a6 a7 a8 a9 aa ab  |................|
000000a0  ac ad ae af b0 b1 b2 b3  b4 b5 b6 b7 b8 b9 ba bb  |................|
000000b0  bc bd be bf c0 c1 c2 c3  c4 c5 c6 c7 c8 c9 ca cb  |................|
000000c0  cc cd ce cf d0 d1 d2 d3  d4 d5 d6 d7 d8 d9 da db  |................|
000000d0  dc dd de df e0 e1 e2 e3  e4 e5 e6 e7 e8 e9 ea eb  |................|
000000e0  ec ed ee ef f0 f1 f2 f3  f4 f5 f6 f7 f8 f9 fa fb  |................|
000000f0  fc fd fe ff 22 0a 7d 0a                           |....".}.|
000000f8
$ base64 -b 72 <lolcat.conf
b2JqZWN0IEhvc3QgImxvbGNhdCIgewoJY2hlY2tfY29tbWFuZCA9ICJob3N0YWxpdmUiCgl2
ZXJzaW9uID0gMTU0MzU3MDY1Ny4zMjAzMDYKCXpvbmUgPSAiQ0VOU09SRUQiCgl2YXJzLmZv
b2JhciA9ICKAgYKDhIWGh4iJiouMjY6PkJGSk5SVlpeYmZqbnJ2en6ChoqOkpaanqKmqq6yt
rq+wsbKztLW2t7i5uru8vb6/wMHCw8TFxsfIycrLzM3Oz9DR0tPU1dbX2Nna29zd3t/g4eLj
5OXm5+jp6uvs7e7v8PHy8/T19vf4+fr7/P3+/yIKfQo=
$ curl -k -H 'Accept: application/json' "$(./url-cluster.sh)/v1/objects/hosts/lolcat" |hexdump -C # v2.10.2
(...)
00000900  2c 22 76 61 72 73 22 3a  7b 22 66 6f 6f 62 61 72  |,"vars":{"foobar|
00000910  22 3a 22 80 81 82 83 84  85 86 87 88 89 8a 8b 8c  |":".............|
00000920  8d 8e 8f 90 91 92 93 94  95 96 97 98 99 9a 9b 9c  |................|
00000930  9d 9e 9f a0 a1 a2 a3 a4  a5 a6 a7 a8 a9 aa ab ac  |................|
00000940  ad ae af b0 b1 b2 b3 b4  b5 b6 b7 b8 b9 ba bb bc  |................|
00000950  bd be bf c0 c1 c2 c3 c4  c5 c6 c7 c8 c9 ca cb cc  |................|
00000960  cd ce cf d0 d1 d2 d3 d4  d5 d6 d7 d8 d9 da db dc  |................|
00000970  dd de df e0 e1 e2 e3 e4  e5 e6 e7 e8 e9 ea eb ec  |................|
00000980  ed ee ef f0 f1 f2 f3 f4  f5 f6 f7 f8 f9 fa fb fc  |................|
00000990  fd fe ff 22 7d 2c 22 76  65 72 73 69 6f 6e 22 3a  |..."},"version":|
(...)
$ curl -k -H 'Accept: application/json' "$(./url-cluster.sh)/v1/objects/hosts/lolcat" |hexdump -C # this PR
(...)
00000900  2c 22 76 61 72 73 22 3a  7b 22 66 6f 6f 62 61 72  |,"vars":{"foobar|
00000910  22 3a 22 ef bf bd ef bf  bd ef bf bd ef bf bd ef  |":".............|
00000920  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
00000930  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
00000940  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
00000950  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
00000960  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
00000970  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
00000980  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
00000990  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
000009a0  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
000009b0  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
000009c0  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
000009d0  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
000009e0  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
000009f0  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
00000a00  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
00000a10  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
00000a20  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
00000a30  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
00000a40  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
00000a50  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
00000a60  ef bf bd ef bf bd ef bf  bd ef bf bd ef bf bd ef  |................|
00000a70  bf bd ef bf bd ef bf bd  ef bf bd ef bf bd ef bf  |................|
00000a80  bd ef bf bd ef bf bd ef  bf bd ef bf bd ef bf bd  |................|
00000a90  ef bf bd 22 7d 2c 22 76  65 72 73 69 6f 6e 22 3a  |..."},"version":|
(...)

@dnsmichi
Copy link
Contributor

I will be working on #6684 soon and replace the entire JSON library causing memory leaks. When at it, I'll test this PR too.

@dnsmichi dnsmichi self-assigned this Feb 11, 2019
@Al2Klimov Al2Klimov removed the request for review from dnsmichi March 15, 2019 10:33
@Al2Klimov Al2Klimov closed this Mar 15, 2019
@Al2Klimov Al2Klimov deleted the bugfix/non-utf-8-broken-json-4703 branch March 15, 2019 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
TBD To be defined - We aren't certain about this yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[dev.icinga.com #12829] non-UTF-8 characters creates broken JSON
2 participants