[SPARK-19319][SparkR]:SparkR Kmeans summary returns error when the cluster size doesn't equal to k #16666
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #71750 has finished for PR 16666 at commit
|
|
ah - does bisecting kmeans have the same behavior? |
felixcheung
reviewed
Jan 21, 2017
R/pkg/R/mllib_clustering.R
Outdated
| #' (cluster centers of the transformed data). | ||
| #' \code{size} (number of data points in each cluster), \code{cluster} | ||
| #' (cluster centers of the transformed data), and \code{clusterSize} | ||
| #' (the actual number of cluster centers. When using initMode = "random", |
There was a problem hiding this comment.
OK. I will add it. For bisecting kmeans, I haven't found a case like this. This case only occurs when initMode is random and this behavior was due to one fix to kmeans implementation.
|
Test build #71826 has started for PR 16666 at commit |
|
Jenkins, retest this please. |
|
Test build #71864 has finished for PR 16666 at commit
|
|
LGTM |
|
ping @felixcheung |
|
Test build #72207 has finished for PR 16666 at commit
|
|
Test build #72214 has finished for PR 16666 at commit
|
|
merged, thanks! |
|
I will backport it soon. Thanks! |
asfgit
pushed a commit
that referenced
this pull request
Feb 12, 2017
…or when the cluster size doesn't equal to k ## What changes were proposed in this pull request? Backport fix of #16666 ## How was this patch tested? Backport unit tests Author: [email protected] <[email protected]> Closes #16761 from wangmiao1981/kmeansport.
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
…uster size doesn't equal to k
## What changes were proposed in this pull request
When Kmeans using initMode = "random" and some random seed, it is possible the actual cluster size doesn't equal to the configured `k`.
In this case, summary(model) returns error due to the number of cols of coefficient matrix doesn't equal to k.
Example:
> col1 <- c(1, 2, 3, 4, 0, 1, 2, 3, 4, 0)
> col2 <- c(1, 2, 3, 4, 0, 1, 2, 3, 4, 0)
> col3 <- c(1, 2, 3, 4, 0, 1, 2, 3, 4, 0)
> cols <- as.data.frame(cbind(col1, col2, col3))
> df <- createDataFrame(cols)
>
> model2 <- spark.kmeans(data = df, ~ ., k = 5, maxIter = 10, initMode = "random", seed = 22222, tol = 1E-5)
>
> summary(model2)
Error in `colnames<-`(`*tmp*`, value = c("col1", "col2", "col3")) :
length of 'dimnames' [2] not equal to array extent
In addition: Warning message:
In matrix(coefficients, ncol = k) :
data length [9] is not a sub-multiple or multiple of the number of rows [2]
Fix: Get the actual cluster size in the summary and use it to build the coefficient matrix.
## How was this patch tested?
Add unit tests.
Author: [email protected] <[email protected]>
Closes apache#16666 from wangmiao1981/kmeans.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request
When Kmeans using initMode = "random" and some random seed, it is possible the actual cluster size doesn't equal to the configured
k.In this case, summary(model) returns error due to the number of cols of coefficient matrix doesn't equal to k.
Example:
Fix: Get the actual cluster size in the summary and use it to build the coefficient matrix.
How was this patch tested?
Add unit tests.