fix: /query rest endpoint should return valid JSON (BREAKING CHANGE)

[big-andy-coates]

Description

fixes #3408 and, I think, Grafana's issues with our /query response

Previously, the RESTful /query endpoint was returning invalid json. For example:

{"row":{"columns":["USER_1","PAGE_1",1,"1"], "errorMessage": null, "finalMessage": null,"terminal":true}}}
{"row":{"columns":["USER_2","PAGE_2",2,"2"], "errorMessage": null, "finalMessage": null,"terminal":true}}}
{"row": null, "errorMessage": null, "finalMessage":"Limit Reached","terminal":true}}"

Each line contained a valid JSON object, but the payload as a whole was invalid and the parsing fails if a string contains an embedded new line character.

The payload as a whole is now a valid JSON document, being a JSON array of rows, and can handle strings with embedded new line characters.
In addition, I've hidden null fields and the 'terminal' field, which was never supposed to be in the JSON. The output now looks like:

[{"row":{"columns":["USER_1","PAGE_1",1,"1"]}},
{"row":{"columns":["USER_2","PAGE_2",2,"2"]}},
{"finalMessage":"Limit Reached"}]"

Note the opening [ on the first row, the trailing , each row and the final ] on the last row.

Of course, for infinite queries, i.e. push queries with no termination/limit clause, the client will never see the terminal ]. However, there are plenty of JSON parsers out there that will happily process such a stream of JSON, e.g. the Jackson streaming API.

Note: this PR fixes the JSON for both the existing push queries and the pull queries added in #3820.

The CLI is backwards compatible with older versions of the server, though it won't output column headings from older versions.

BREAKING CHANGE: the response from the RESTful API for push queries has changed: it is now a valid JSON document containing a JSON array, where each element is JSON object containing either a row of data, an error message, or a final message. The terminal field has been removed.

Testing done

tested added, mvn test and manual testing of CLI

Reviewer checklist

• Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
• Ensure relevant issues are linked (description should include text like "Fixes #")

[vpapavas]

Thank you @big-andy-coates! One high-level question, what happens if the query does not have the LIMIT clause? Is a response still a valid json object?

[vpapavas]

LGTM after the discussion we had in the meeting. Please get one more +1.

[vinothchandar]

LGTM.

[big-andy-coates]

Thank you @big-andy-coates! One high-level question, what happens if the query does not have the LIMIT clause? Is a response still a valid json object?

It's a valid question! The answer is that isn't not possible to reply with a valid JSON response if the response never ends.

The HTTP standard says that a response with media type JSON should contain a single well formed JSON document or value. This is not possible for infinite result sets. Hence #3816 to look at redesigning our API.

That said, the returned JSON is not invalid, it's just never ending / never completes. There are libraries that can happily parse such a stream of JSON, e.g. Jackson has a streaming API.