[SPARK-23062][SQL] Improve EXCEPT documentation #20254
Conversation
|
Done, thanks for pointing that out! |
R/pkg/R/DataFrame.R
Outdated
| @@ -2873,6 +2873,7 @@ setMethod("intersect", | |||
| #' @rdname except | |||
| #' @export | |||
| #' @note except since 1.4.0 | |||
| #' @note behaviour changed from \code{EXCEPT ALL} to \code{EXCEPT DISTINCT} in 2.0. | |||
There was a problem hiding this comment.
Ur.. I think we have use @note for version specification in SparkR. Just adding
Note: blabla
should be fine like other places.
There was a problem hiding this comment.
I don't mind it either way, but to note:
- r doc order and whitespace is significant, if you use
#' Note:you must put it after L2856, if you put an extra#'ie. empty line that it becomes theDetailssection, which might be the right place; see http://spark.apache.org/docs/latest/api/R/awaitTermination.html
There was a problem hiding this comment.
ie.
#' but not in another SparkDataFrame. This is equivalent to \code{EXCEPT DISTINCT} in SQL.
#'
#' Note: Before Spark 2.0.0, the behavior was equivalent to `EXCEPT ALL` in SQL.
#'
#' @param x a SparkDataFrame.
There was a problem hiding this comment.
This is wrong. See my previous comment.
python/pyspark/sql/dataframe.py
Outdated
| This is equivalent to `EXCEPT` in SQL. | ||
| This is equivalent to `EXCEPT DISTINCT` in SQL. | ||
|
|
||
| (Note: Before Spark 2.0, the behavior was equivalent to `EXCEPT ALL` in SQL.) |
There was a problem hiding this comment.
In PySpark, we can use .. note:: . This makes the doc pretty :).
There was a problem hiding this comment.
Actually, before 2.0, it is not equivalent to EXCEPT ALL. For details, see the PR: #12736
|
Test build #86067 has finished for PR 20254 at commit
|
|
Test build #86068 has finished for PR 20254 at commit
|
|
Also, cc @gatorsmile . |
|
I see, from reading that PR I think perhaps we should reference migration guide in sql programming guide instead of putting the whole description here.
|
|
Yeah, we should document the behavior changes, but that was just a bug fix for 100% following the semantics of ANSI-SQL |
|
@henryr could you update this PR to only include EXCEPT DISTINCT without the notes |
|
Just FYI, in ANSI SQL, |
## What changes were proposed in this pull request? Make the default behavior of EXCEPT (i.e. EXCEPT DISTINCT) more explicit in the documentation.
|
Thanks all for the pointers and feedback! I've removed the references to the behavior before 2.0, and now the changes just make it explicit that this is |
|
@henryr Since Spark 2.3, Spark SQL documents all the behavior changes in Migration Guides. Hopefully, this can help our end users. |
|
LGTM |
|
Test build #86211 has finished for PR 20254 at commit
|
## What changes were proposed in this pull request? Make the default behavior of EXCEPT (i.e. EXCEPT DISTINCT) more explicit in the documentation, and call out the change in behavior from 1.x. Author: Henry Robinson <[email protected]> Closes #20254 from henryr/spark-23062. (cherry picked from commit 1f3d933) Signed-off-by: gatorsmile <[email protected]>
What changes were proposed in this pull request?
Make the default behavior of EXCEPT (i.e. EXCEPT DISTINCT) more
explicit in the documentation, and call out the change in behavior
from 1.x.