[SPARK-24217][ML]Power Iteration Clustering is not displaying cluster indices corresponding to some vertices

[shahidki31]

What changes were proposed in this pull request?

1. Currently PIC in ML displays cluster indices of nodes in the ID column. If some of the nodes there in the neighboring column, but not there in the ID column, It will not display the cluster indices corresponding to that node.

As per the definition of PIC clustering, given in the code,

PIC takes an affinity matrix between items (or vertices) as input. An affinity matrix
is a symmetric matrix whose entries are non-negative similarities between items.
PIC takes this matrix (or graph) as an adjacency matrix. Specifically, each input row includes:

• idCol: vertex ID
• neighborsCol: neighbors of vertex in idCol
• similaritiesCol: non-negative weights (similarities) of edges between the vertex
  in idCol and each neighbor in neighborsCol

• "PIC returns a cluster assignment for each input vertex." It appends a new column predictionCol
  containing the cluster assignment in [0,k) for each row (vertex).

2. We should display prediction and id corresponding to all the nodes. So, instead of the join operation to the existing dataFrame, we can directly return the prediction-id dataFrame corresponding to all the nodes.
   For the same input in spark.ml and spark.mllib, spark.mllib giving cluster id for all the vertices.

How was this patch tested?

UT

Please review http://spark.apache.org/contributing.html before opening a pull request.

[shahidki31]

Based on the comments in the JIRA, ( https://issues.apache.org/jira/browse/SPARK-24217), I am closing the PR.

[AmplabJenkins]

Can one of the admins verify this patch?

[shahidki31]

Closing the PR due to the discussions in the JIRA, https://issues.apache.org/jira/browse/SPARK-15784 and the PR #21493