[SPARK-4130][MLlib] Fixing libSVM parser bug with extra whitespace

This simple patch filters out extra whitespace entries. Author: Joseph E. Gonzalez <[email protected]> Author: Joey <[email protected]> Closes #2996 from jegonzal/loadLibSVM and squashes the following commits: e0227ab [Joey] improving readability e028e84 [Joseph E. Gonzalez] fixing whitespace bug in loadLibSVMFile when parsing libSVM files
apache · Oct 30, 2014 · c7ad085 · c7ad085
1 parent 6db3157
commit c7ad085
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
@@ -76,7 +76,7 @@ object MLUtils {
       .map { line =>
         val items = line.split(' ')
         val label = items.head.toDouble
-        val (indices, values) = items.tail.map { item =>
+        val (indices, values) = items.tail.filter(_.nonEmpty).map { item =>
           val indexAndValue = item.split(':')
           val index = indexAndValue(0).toInt - 1 // Convert 1-based indices to 0-based.
           val value = indexAndValue(1).toDouble