From 35a2cd62a4bdb2ec2835540e0c374f64f701faf0 Mon Sep 17 00:00:00 2001 From: Tim Fox Date: Mon, 10 Feb 2020 09:08:18 +0000 Subject: [PATCH 1/4] Bring klip up to date --- .../klip-15-new-api-and-client.md | 22 +++++++------------ 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/design-proposals/klip-15-new-api-and-client.md b/design-proposals/klip-15-new-api-and-client.md index 0b973dfe8424..54a6184ec691 100644 --- a/design-proposals/klip-15-new-api-and-client.md +++ b/design-proposals/klip-15-new-api-and-client.md @@ -77,6 +77,10 @@ The API will have the following characteristics: * TLS * Easy to create new clients using the protocol +Please note the parameters are not necessarily exhaustive. The description here is an outline not a detailed +low level design. The low level design will evolve during development, and the web-site API docs +will provide exact and detailed documentation on the endpoints and how to use them. + #### Query streaming The request method will be a POST. @@ -89,8 +93,6 @@ operation (newlines have been added here for the sake of clarity but the real JS ```` { "query": "select * from foo", <----- the SQL of the query to execute -"push": true, <----- if true then push query, else pull query -"limit": 10 <---- If specified return at most this many rows, "properties": { <----- Optional properties for the query "prop1": "val1", "prop2": "val2" @@ -99,9 +101,6 @@ operation (newlines have been added here for the sake of clarity but the real JS ```` -Please note the parameters are not necessarily exhaustive. The description here is an outline not a detailed -low level design. The low level design will evolve during development. - In the case of a successful query ```` @@ -109,7 +108,6 @@ In the case of a successful query "query-id", "xyz123", <---- unique ID of the query, used when terminating the query "columns":["col", "col2", "col3"], <---- the names of the columns "column_types":["BIGINT", "STRING", "BOOLEAN"] <---- The types of the columns -"row_count": 101 <---- The number of rows - only set in case of pull query } ```` @@ -158,7 +156,7 @@ operation (newlines have been added for clarity, the real JSON must not contain ```` { "stream": "my-stream" <----- The name of the KSQL stream to insert into -"acks": true <----- If true then a stream of acks will be returned in the response +"requiresAcks": true <----- If true then a stream of acks will be returned in the response } ```` @@ -209,12 +207,8 @@ For this reason we do not provide query results (or accept streams of inserts) b JSON object. If we did so we would force users to find and use a streaming JSON parser in order to parse the results as they arrive. If no parser is available on their platform they would be forced to parse the entire result set as a single JSON object in memory - this might not be feasible or -desirable due to memory and latency constraints. Moreover, in the world of streaming it's quite common -to want to pipe the stream from one place to another without looking into it. In that case it's very -inefficient to deserialize the bytes as JSON simply to serialize them back to bytes to write them -to the output stream. For these reasons the results, by default, are sent as a set of JSON arrays -delimited by newline. Newlines are very easy to parse by virtually every target platform without -having to rely on specialist libraries. +desirable due to memory and latency constraints. Newlines are very easy to parse by virtually every +target platform without having to rely on specialist libraries. There are, however, some use cases where we can guarantee the results of the query are small, e.g. when using a limit clause. In this case, the more general streaming use case degenerates into an RPC @@ -236,7 +230,7 @@ We will migrate the existing Jetty specific plug-ins to Vert.x The new server API will be implemented using Vert.x -The implementation will be designed in a reactive / non blocking way in order to provide the best performance / scalability characteristics with +The implementation will be designed in a reactive / non-blocking way in order to provide the best performance / scalability characteristics with low resource usage. This will also influence the overall threading model of the server and position us with a better, more scalability internal server architecture that will help future proof the ksqlDB server. From 185a28de0bc67ba55e5aba39d03dcc073d1eb841 Mon Sep 17 00:00:00 2001 From: Tim Fox Date: Wed, 12 Feb 2020 07:52:33 +0000 Subject: [PATCH 2/4] more updates --- .../klip-15-new-api-and-client.md | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/design-proposals/klip-15-new-api-and-client.md b/design-proposals/klip-15-new-api-and-client.md index 54a6184ec691..78312695f1c5 100644 --- a/design-proposals/klip-15-new-api-and-client.md +++ b/design-proposals/klip-15-new-api-and-client.md @@ -88,12 +88,13 @@ The request method will be a POST. Requests will be made to a specific URL, e.g. "/query-stream" (this can be configurable) The body of the request is a JSON object UTF-8 encoded as text, containing the arguments for the -operation (newlines have been added here for the sake of clarity but the real JSON must not contain newlines) +operation (newlines have been added here for the sake of clarity but the real JSON must not contain + unescaped newlines) ```` { -"query": "select * from foo", <----- the SQL of the query to execute -"properties": { <----- Optional properties for the query +"sql": "select * from foo", <----- the SQL of the query to execute +"properties": { <----- Optional properties for the query "prop1": "val1", "prop2": "val2" } @@ -105,9 +106,9 @@ In the case of a successful query ```` { -"query-id", "xyz123", <---- unique ID of the query, used when terminating the query -"columns":["col", "col2", "col3"], <---- the names of the columns -"column_types":["BIGINT", "STRING", "BOOLEAN"] <---- The types of the columns +"queryId", "xyz123", <---- unique ID of the query, used when terminating the query +"columnNames":["col", "col2", "col3"], <---- the names of the columns +"columnTypes":["BIGINT", "STRING", "BOOLEAN"] <---- The types of the columns } ```` @@ -131,14 +132,14 @@ Push queries can be explicitly terminated by the client by making a request to t The request method will be a POST. -Requests will be made to a specific URL, e.g. "/query-terminate" (this can be configurable) +Requests will be made to a specific URL, e.g. "/query-close" (this can be configurable) The body of the request is a JSON object UTF-8 encoded as text, containing the arguments for the operation (newlines have been added here for the sake of clarity but the real JSON must not contain newlines) ```` { -"query-id": "xyz123", <----- the ID of the query to terminate +"queryId": "xyz123", <----- the ID of the query to terminate } ```` @@ -155,8 +156,7 @@ operation (newlines have been added for clarity, the real JSON must not contain ```` { -"stream": "my-stream" <----- The name of the KSQL stream to insert into -"requiresAcks": true <----- If true then a stream of acks will be returned in the response +"target": "my-stream" <----- The name of the KSQL stream to insert into } ```` @@ -174,23 +174,33 @@ Each JSON object will be separated by a new line. To terminate the insert stream the client should end the request. -If acks are requested then the response will be written to the response when each row has been +Acks will be written to the response when each row has been successfully committed to the underlying topic. Rows are committed in the order they are provided. -Each ack in the response is just an empty JSON object, separated by newlines: +Each ack in the response is a JSON object, separated by newlines: ```` -{} -{} -{} -{} +{"status":"ok","seq":0} +{"status":"ok","seq":2} +{"status":"ok","seq":1} +{"status":"ok","seq":3} ```` +A successful ack will contain a field `status` with value `ok`. + +All ack responses also contain a field `seq` with a 64 bit signed integer value. This number +corresponds to the sequence of the send on the request. The first send has sequence `0`, the second +`1`, the third `3`, etc. It allows the client to correlate the ack to the corresponding send. + +In case of error, an error response (see below) will be sent. For an error response for a send, the +`seq` field will also be included. + #### Errors Apropriate status codes will be returned from failed requests. The response will also contain JSON with further information on the error: { + "status": "error" "error_code": "message": } From 88e322a396270e1523faf6edfcaef22a9868cb3b Mon Sep 17 00:00:00 2001 From: Tim Fox Date: Wed, 12 Feb 2020 12:57:53 +0000 Subject: [PATCH 3/4] tweak --- design-proposals/klip-15-new-api-and-client.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/design-proposals/klip-15-new-api-and-client.md b/design-proposals/klip-15-new-api-and-client.md index 78312695f1c5..4b694fcb2969 100644 --- a/design-proposals/klip-15-new-api-and-client.md +++ b/design-proposals/klip-15-new-api-and-client.md @@ -188,12 +188,14 @@ Each ack in the response is a JSON object, separated by newlines: A successful ack will contain a field `status` with value `ok`. All ack responses also contain a field `seq` with a 64 bit signed integer value. This number -corresponds to the sequence of the send on the request. The first send has sequence `0`, the second +corresponds to the sequence of the insert on the request. The first send has sequence `0`, the second `1`, the third `3`, etc. It allows the client to correlate the ack to the corresponding send. In case of error, an error response (see below) will be sent. For an error response for a send, the `seq` field will also be included. +Please note that acks can be returned in a different sequence to which the inserts were submitted. + #### Errors Apropriate status codes will be returned from failed requests. The response will also contain JSON From 7639985272e6fe9fb1715d379eaa54f21408f80b Mon Sep 17 00:00:00 2001 From: Tim Fox Date: Fri, 14 Feb 2020 07:42:03 +0000 Subject: [PATCH 4/4] tweaks --- design-proposals/klip-15-new-api-and-client.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design-proposals/klip-15-new-api-and-client.md b/design-proposals/klip-15-new-api-and-client.md index 4b694fcb2969..1f240c8467ce 100644 --- a/design-proposals/klip-15-new-api-and-client.md +++ b/design-proposals/klip-15-new-api-and-client.md @@ -132,7 +132,7 @@ Push queries can be explicitly terminated by the client by making a request to t The request method will be a POST. -Requests will be made to a specific URL, e.g. "/query-close" (this can be configurable) +Requests will be made to a specific URL, e.g. "/close-query" The body of the request is a JSON object UTF-8 encoded as text, containing the arguments for the operation (newlines have been added here for the sake of clarity but the real JSON must not contain newlines) @@ -189,7 +189,7 @@ A successful ack will contain a field `status` with value `ok`. All ack responses also contain a field `seq` with a 64 bit signed integer value. This number corresponds to the sequence of the insert on the request. The first send has sequence `0`, the second -`1`, the third `3`, etc. It allows the client to correlate the ack to the corresponding send. +`1`, the third `2`, etc. It allows the client to correlate the ack to the corresponding send. In case of error, an error response (see below) will be sent. For an error response for a send, the `seq` field will also be included.