[SPARK-19584] [SS] [DOCS] update structured streaming documentation around batch mode #16918

tcondie · 2017-02-14T00:06:20Z

What changes were proposed in this pull request?

Revision to structured-streaming-kafka-integration.md to reflect new Batch query specification and options.

@zsxwing @tdas

Please review http://spark.apache.org/contributing.html before opening a pull request.

SparkQA · 2017-02-14T00:33:27Z

Test build #72840 has finished for PR 16918 at commit 081ea24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-14T00:35:02Z

Test build #72839 has finished for PR 16918 at commit debe111.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing

Overall looks good. Left some nits

zsxwing · 2017-02-14T00:27:33Z

docs/structured-streaming-kafka-integration.md

@@ -119,6 +119,122 @@ ds3.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
 </div>
 </div>

+### Creating a Kafka Source Batch
+


It's better to add a sentence here before the codes, such as If you have a use case that is better suited to batch processing, you can create an Dataset/DataFrame for a defined range of offsets.

zsxwing · 2017-02-14T00:32:22Z

docs/structured-streaming-kafka-integration.md

+  </td>
+  <td>latest</td>
+  <td>batch only</td>
+  <td>The end point when a batch query is started, either "latest" which is just from the latest


nit The end point when a batch query is started , either "latest" which is just from the latest -> The end point when a batch query is ended, either "latest" which is just referred to the latest

nit: a json string specifying a starting offset for each TopicPartition. -> a json string specifying an ending offset for each TopicPartition.

zsxwing · 2017-02-14T00:51:02Z

docs/structured-streaming-kafka-integration.md

+  .format("kafka")
+  .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
+  .option("subscribe", "topic1")
+  .load()


nit: missing ";"

zsxwing · 2017-02-14T00:51:11Z

docs/structured-streaming-kafka-integration.md

+  .option("subscribe", "topic1,topic2")
+  .option("startingOffsets", "{\"topic1\":{\"0\":23,\"1\":-2},\"topic2\":{\"0\":-2}}")
+  .option("endingOffsets", "{\"topic1\":{\"0\":50,\"1\":-1},\"topic2\":{\"0\":-1}}")
+  .load()


nit: missing ";"

zsxwing · 2017-02-14T00:51:15Z

docs/structured-streaming-kafka-integration.md

+  .option("startingOffsets", "{\"topic1\":{\"0\":23,\"1\":-2},\"topic2\":{\"0\":-2}}")
+  .option("endingOffsets", "{\"topic1\":{\"0\":50,\"1\":-1},\"topic2\":{\"0\":-1}}")
+  .load()
+ds2.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")


nit: missing ";"

zsxwing · 2017-02-14T00:55:22Z

docs/structured-streaming-kafka-integration.md

+{% highlight python %}
+
+# Subscribe to 1 topic defaults to the earliest and latest offsets
+ds1 = spark


You need to add \ to tell the Python compiler not starting a new line.

SparkQA · 2017-02-14T02:31:44Z

Test build #72843 has finished for PR 16918 at commit 28af040.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2017-02-14T05:04:39Z

docs/structured-streaming-kafka-integration.md

+
+# Subscribe to 1 topic defaults to the earliest and latest offsets
+ds1 = spark \
+  .read


You need to all \ to all lines.

SparkQA · 2017-02-14T18:55:18Z

Test build #72887 has finished for PR 16918 at commit f5318e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2017-02-14T19:13:09Z

@tcondie There is a typo about fetchOffset.numRetries in this page: Number of times to retry before giving up **fatch** Kafka **latest** offsets. -> Number of times to retry before giving up **fetching** Kafka **** offsets. Could you also fix it since you are touching this file?

tdas · 2017-02-14T19:25:44Z

docs/structured-streaming-kafka-integration.md

@@ -152,7 +270,7 @@ Each row in the source has the following schema:
 </tr>
 </table>

-The following options must be set for the Kafka source.
+The following options must be set for the Kafka source (streaming and batch).


nit: Kafka source for both batch and streaming queries.

tdas · 2017-02-14T19:26:12Z

docs/structured-streaming-kafka-integration.md

@@ -187,50 +305,68 @@ The following options must be set for the Kafka source.
 The following configurations are optional:

 <table class="table">
-<tr><th>Option</th><th>value</th><th>default</th><th>meaning</th></tr>
+<tr><th>Option</th><th>value</th><th>default</th><th>mode</th><th>meaning</th></tr>


rather than "mode", how about "query type". just the term "mode" does not mean much, but "query type" says something.

tdas · 2017-02-14T19:29:29Z

docs/structured-streaming-kafka-integration.md

 <tr>
  <td>startingOffsets</td>
-  <td>earliest, latest, or json string
+  <td>earliest, latest (streaming only), or json string


can you put the possible values in quotes. e.g.

"earliest", "latest" (only for streaming) or json string """ {"topicA":{"0":23,"1":-1},"topicB":{"0":-2}} """

tdas · 2017-02-14T19:29:50Z

docs/structured-streaming-kafka-integration.md

  <td>The start point when a query is started, either "earliest" which is from the earliest offsets,
  "latest" which is just from the latest offsets, or a json string specifying a starting offset for
  each TopicPartition.  In the json, -2 as an offset can be used to refer to earliest, -1 to latest.
-  Note: This only applies when a new Streaming query is started, and that resuming will always pick
-  up from where the query left off. Newly discovered partitions during a query will start at
+  Note: For Batch, latest (either implicitly or by using -1 in json) is not allowed.


nit: For batch queries...

tdas · 2017-02-14T19:30:39Z

docs/structured-streaming-kafka-integration.md

  {"topicA":{"0":23,"1":-1},"topicB":{"0":-2}}
  </td>
-  <td>latest</td>
+  <td>streaming=latest, batch=earliest</td>


same as above, put the value quotes. its clearer if
"latest" for streaming,
"earliest" for batch

tdas · 2017-02-14T19:32:12Z

docs/structured-streaming-kafka-integration.md

-  Note: This only applies when a new Streaming query is started, and that resuming will always pick
-  up from where the query left off. Newly discovered partitions during a query will start at
+  Note: For Batch, latest (either implicitly or by using -1 in json) is not allowed.
+  For Streaming, this only applies when a new Streaming query is started, and that resuming will


nit: For streaming queries, this only ... when a new query is stasrted

SparkQA · 2017-02-14T22:07:16Z

Test build #72895 has finished for PR 16918 at commit 5bf32f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2017-02-14T22:30:21Z

docs/structured-streaming-kafka-integration.md

  <td>failOnDataLoss</td>
  <td>true or false</td>
  <td>true</td>
-  <td>Whether to fail the query when it's possible that data is lost (e.g., topics are deleted, or 
+  <td>streaming only</td>
+  <td>Whether to fail the query when it's possible that data is lost (e.g., topics are deleted, or


nit: the streaming query
also could you add, what is the behavior for batch queries? that is the batch query will always fail if it fails to read any data.

tdas · 2017-02-14T22:31:34Z

docs/structured-streaming-kafka-integration.md

  <td>The start point when a query is started, either "earliest" which is from the earliest offsets,
  "latest" which is just from the latest offsets, or a json string specifying a starting offset for
  each TopicPartition.  In the json, -2 as an offset can be used to refer to earliest, -1 to latest.
-  Note: This only applies when a new Streaming query is started, and that resuming will always pick
-  up from where the query left off. Newly discovered partitions during a query will start at
+  Note: For Batch queries, latest (either implicitly or by using -1 in json) is not allowed.


nit: why is B and S in Batch queries and Streaming queries randomly caps? I dont think "batch query" is a proper name of any API, and neither is "streaming query", unless you are referring the specific class "StreamingQuery".

Agreed. The previous version had "S", so I wasn't sure if this was a convention or not. I'll make the correction.

tdas · 2017-02-14T22:32:23Z

@tcondie this is close and very good. just one more nit. otherwise LGTM.

SparkQA · 2017-02-14T23:41:30Z

Test build #72904 has finished for PR 16918 at commit 8983b8d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-15T01:05:09Z

Test build #72907 has finished for PR 16918 at commit d55ba09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2017-02-15T02:49:14Z

LGTM. merging now.

…und batch mode ## What changes were proposed in this pull request? Revision to structured-streaming-kafka-integration.md to reflect new Batch query specification and options. zsxwing tdas Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Tyson Condie <[email protected]> Closes #16918 from tcondie/kafka-docs. (cherry picked from commit 447b2b5) Signed-off-by: Tathagata Das <[email protected]>

…und batch mode ## What changes were proposed in this pull request? Revision to structured-streaming-kafka-integration.md to reflect new Batch query specification and options. zsxwing tdas Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Tyson Condie <[email protected]> Closes apache#16918 from tcondie/kafka-docs.

tcondie added 2 commits February 13, 2017 16:01

update structured streaming documentation around batch mode

debe111

update

081ea24

zsxwing requested changes Feb 14, 2017

View reviewed changes

address comments from @zsxwing

28af040

zsxwing reviewed Feb 14, 2017

View reviewed changes

updatedd @zsxwing

f5318e0

tdas reviewed Feb 14, 2017

View reviewed changes

update addressing comments from @zsxwing and @tdas

5bf32f4

tdas reviewed Feb 14, 2017

View reviewed changes

revised based on feedback from @tdas

8983b8d

revision @tdas

d55ba09

asfgit closed this in 447b2b5 Feb 15, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19584] [SS] [DOCS] update structured streaming documentation around batch mode #16918

[SPARK-19584] [SS] [DOCS] update structured streaming documentation around batch mode #16918

tcondie commented Feb 14, 2017 •

edited

Loading

SparkQA commented Feb 14, 2017

SparkQA commented Feb 14, 2017

zsxwing left a comment

zsxwing Feb 14, 2017

zsxwing Feb 14, 2017

zsxwing Feb 14, 2017

zsxwing Feb 14, 2017

zsxwing Feb 14, 2017

zsxwing Feb 14, 2017

zsxwing Feb 14, 2017

SparkQA commented Feb 14, 2017

zsxwing Feb 14, 2017

SparkQA commented Feb 14, 2017

zsxwing commented Feb 14, 2017

tdas Feb 14, 2017

tdas Feb 14, 2017 •

edited

Loading

tdas Feb 14, 2017

tdas Feb 14, 2017

tdas Feb 14, 2017

tdas Feb 14, 2017

SparkQA commented Feb 14, 2017

tdas Feb 14, 2017

tdas Feb 14, 2017

tcondie Feb 14, 2017 •

edited

Loading

tdas commented Feb 14, 2017

SparkQA commented Feb 14, 2017

SparkQA commented Feb 15, 2017

tdas commented Feb 15, 2017

[SPARK-19584] [SS] [DOCS] update structured streaming documentation around batch mode #16918

[SPARK-19584] [SS] [DOCS] update structured streaming documentation around batch mode #16918

Conversation

tcondie commented Feb 14, 2017 • edited Loading

What changes were proposed in this pull request?

SparkQA commented Feb 14, 2017

SparkQA commented Feb 14, 2017

zsxwing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 14, 2017

Choose a reason for hiding this comment

SparkQA commented Feb 14, 2017

zsxwing commented Feb 14, 2017

Choose a reason for hiding this comment

tdas Feb 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcondie Feb 14, 2017 • edited Loading

Choose a reason for hiding this comment

tdas commented Feb 14, 2017

SparkQA commented Feb 14, 2017

SparkQA commented Feb 15, 2017

tdas commented Feb 15, 2017

tcondie commented Feb 14, 2017 •

edited

Loading

tdas Feb 14, 2017 •

edited

Loading

tcondie Feb 14, 2017 •

edited

Loading