[SPARK-16060][SQL] Support Vectorized ORC Reader #19943

dongjoon-hyun · 2017-12-11T22:52:52Z

What changes were proposed in this pull request?

This PR adds an ORC columnar-batch reader to native OrcFileFormat. Since both Spark ColumnarBatch and ORC RowBatch are used together, it is faster than the current Spark implementation. This replaces the prior PR, #17924.

Also, this PR adds OrcReadBenchmark to show the performance improvement.

How was this patch tested?

Pass the existing test cases.

ghost · 2017-12-11T23:13:00Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+
+              case TimestampType =>
+                val data = fromColumn.asInstanceOf[TimestampColumnVector]
+                toColumn.appendLongs(batchSize, data.time(0) * 1000L + data.nanos(0) / 1000L)


This raw calculation here should be encapsulated in a utilities method with a few comments explaining what is happening (i.e. the conversion from TimestampColumnVector's java.sql.Timestamp representation to Spark's). It is repeated (so to speak several times in this method/

ghost · 2017-12-11T23:18:07Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+                }
+
+              case DecimalType.Fixed(precision, scale) =>
+                val d = fromColumn.asInstanceOf[DecimalColumnVector].vector(0)


Similarly, the conversion from DecimalColumnVector's HiveDecimalWritable to Spark's representation should be encapsulated in a utilities method and explained. Note that HiveDecimalWritable does provide the possibility of converting that are faster than going first to BigDecimal. Having one version of the code would make that easier in the future..

It's refactored first.

ghost · 2017-12-11T23:31:10Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+                throw new UnsupportedOperationException(s"Unsupported Data Type: $dt")
+            }
+          }
+        } else if (!field.nullable || fromColumn.noNulls) {


I'm not sure I get how this if stmt works right.

Could you elaborate your concern on this?

While I get your schema says the column has no nulls (!field.nullable) that isn't a guarantee that the column returned by the reader is noNulls. There should be an assertion or perhaps better yet an exception thrown for a data error. (My other comment about selectedInUse probably ought to be a data error, too).

This method talks about ORC reader -- yet really this is general Hive to Spark vectorization code and it will in one way or another be generalized and reused. In Hive, there is work to vectorize Parquet fully. We also support vectorized reading of text files and through LLAP.

(Nulls can sneak in when enforcing decimal precision/scale, for example).

Throwing data error exceptions can be nasty -- perhaps setting the value to a known value and a warning might be better. I noticed leveraging of not nullable in the Hive optimizer recently but we will see if there are any practical issues that arise...

Oh, I missed your comment here, @mmccline . Thank you for review!

@mmccline . For this, I'll respect only fromColumn.noNulls here.

ghost · 2017-12-11T23:34:25Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+      if (requestedColIds(i) < 0) {
+        toColumn.appendNulls(batchSize)
+      } else {
+        val fromColumn = batch.cols(requestedColIds(i))


The VectorizedRowBatch processing here is ignoring boolean batch.selectedInUse and int[] batch.selected array logical batch index processing. If you are going to assume batch.selectedInUse is false for batches read from ORC, it ought to be asserted.

Yep. I'll add assertion.

dongjoon-hyun · 2017-12-12T00:04:16Z

Thank you for review, @mmccline !

viirya · 2017-12-12T00:12:05Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -378,6 +378,11 @@ object SQLConf {
    .checkValues(Set("hive", "native"))
    .createWithDefault("native")

+  val ORC_VECTORIZED_READER_ENABLED = buildConf("spark.sql.orc.vectorizedReader.enabled")


parquet config: spark.sql.parquet.enableVectorizedReader. Shall we have similar config name?

Sure, @viirya . Thank you for review.

viirya · 2017-12-12T00:17:25Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala

@@ -139,15 +146,25 @@ class OrcFileFormat
      }
    }

+    val resultSchema = StructType(requiredSchema.fields ++ partitionSchema.fields)
+    val enableVectorizedReader = sparkSession.sessionState.conf.orcVectorizedReaderEnabled &&
+      supportBatch(sparkSession, resultSchema)


Whether enabling vectorized reader is not the same as supporting batch. You can enable vectorized reader but not support batch like ParquetFileFormat does.

Yep. I'll try.

Ur, it seems to make the implementation complicated in order to provide slower version.
If you don't mind, I'll proceed that in a next follow-up PR.

supportBatch already have the condition conf.orcVectorizedReaderEnabled, I think here we can just write

val enableVectorizedReader = supportBatch(sparkSession, resultSchema)

Right. It's fixed.

viirya · 2017-12-12T00:28:55Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala

+
+      val fs = filePath.getFileSystem(conf)
+      val readerOptions = OrcFile.readerOptions(conf).filesystem(fs)
+      val reader = OrcFile.createReader(filePath, readerOptions)


Why extract the creation of reader from requestedColumnIds to here?

The reader is used here, too. This extraction prevents redundant creation of ORC reader.

batchReader.setRequiredSchema( OrcUtils.getFixedTypeDescription(reader.getSchema, dataSchema),

viirya · 2017-12-12T00:29:55Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala

@@ -110,4 +107,21 @@ object OrcUtils extends Logging {
      }
    }
  }
+
+  /**
+   * Return a fixed ORC schema with data schema information, if needed.


Maybe explain what the issue to fix is?

Yep. I added.

SparkQA · 2017-12-12T01:49:42Z

Test build #84733 has finished for PR 19943 at commit 449a0a2.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-12T21:55:13Z

Test build #84785 has finished for PR 19943 at commit 601ab5e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-12T22:13:40Z

Test build #84788 has finished for PR 19943 at commit d302259.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-12-12T23:45:01Z

Retest this please

SparkQA · 2017-12-13T02:43:25Z

Test build #84809 has finished for PR 19943 at commit d302259.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-12-13T02:55:33Z

Hi, @cloud-fan , @gatorsmile , @HyukjinKwon , @viirya , @mmccline .
The PR is ready for review again. Could you take a look?

cloud-fan · 2017-12-13T03:03:21Z

A high-level question: @viirya had a PR to do this by creating a wrapper for ORC columnar batch. The parquet data source pick a different approach that writes the values to Spark columnar batch.

Generally I think the wrapper approach should be faster on pure scan, but may be slower if there is computation after the scan, e.g. aggregate. Do we have a benchmark for it?

dongjoon-hyun · 2017-12-13T03:21:40Z

That's a good point, @cloud-fan . I also agreed to see the result for that before (at the my initial PR.)

First of all, we can compare the results with OrcReadBenchmark.scala in this PR before going with aggregation. Since we have the foundation for that, @viirya can update his PR more easily than before. I love to see the fastest Apache Spark ORC!

cloud-fan · 2017-12-13T04:43:55Z

Also cc @kiszk , this question also applies to the table cache reader. We should think more about using a wrapper or writing to spark column vector.

cloud-fan · 2017-12-22T08:30:42Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+      columnarBatch.close()
+      columnarBatch = null
+    }
+    if (rows != null) {


do we need to close the orc batch?

cloud-fan · 2017-12-22T08:32:05Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+/**
+ * To support vectorization in WholeStageCodeGen, this reader returns ColumnarBatch.
+ */
+private[orc] class OrcColumnarBatchReader extends RecordReader[Void, ColumnarBatch] with Logging {


We need more documents. How will this class be used? Created for each Spark task and reset(call .initialize) for each file split?

Yep. I can add that. But, do you think we can proceed this approach?
Based on the previous advice, I think you prefer @viirya 's approach.

I think it's fine to follow parquet and write data to Spark column vector now. Later we can try the wrapper approach and compare.

Thank you for decision, @cloud-fan . If then, I'll try to update the PR.

henrify · 2017-12-27T04:45:33Z

If i've understood Spark development process correctly, the 2.3 branch cut date is in couple of days, and if this PR doesn't get merged to master real soon, it'll have to wait until 2.4, about 6 months?

@dongjoon-hyun @cloud-fan Considering that the benchmarks show almost order of magnitude improvement in performance, it would be really great to get this in for Spark 2.3, and worry about the details of copy vs wrapper approach later.

Also, as this is anyway opt-in feature that needs to be enabled with config option, merging this shouldn't be "dangerous".. Thanks for your efforts & looking forward to get this PR to production!

kiszk · 2017-12-27T05:32:46Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+                  toColumn.appendByteArray(data.vector(0), data.start(0), data.length(0))
+                  index += 1
+                }
+              case BinaryType =>


Can we merge two cases for StringType and BinaryType into one case?

kiszk · 2017-12-27T05:36:57Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+            case BooleanType =>
+              val data = fromColumn.asInstanceOf[LongColumnVector].vector
+              var index = 0
+              while (index < batchSize) {


Can we use WritableColumn.appendBooleans()?

It's not. data(index) is changed at each iteration.

I thought appendBooleans() can be used since toColumn.appendLongs() is used for LongType. I realized data is Long as you pointed out .

hmm, so the orc batch always return long as value? that seems very inefficient...

anyone knows why ORC does this? cc @mmccline

The reason is that ORC library has LongColumnVector and DoubleColumnVector for implementation simplicity.
All integral types use LongColumnVector and real types(float/double) use DoubleColumnVector. For non-numeric types, there exists corresponding column vectors.

kiszk · 2017-12-27T05:37:08Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+            case ByteType =>
+              val data = fromColumn.asInstanceOf[LongColumnVector].vector
+              var index = 0
+              while (index < batchSize) {


kiszk · 2017-12-27T05:37:15Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+            case ShortType =>
+              val data = fromColumn.asInstanceOf[LongColumnVector].vector
+              var index = 0
+              while (index < batchSize) {


kiszk · 2017-12-27T05:37:37Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+            case IntegerType | DateType =>
+              val data = fromColumn.asInstanceOf[LongColumnVector].vector
+              var index = 0
+              while (index < batchSize) {


Unfortunately, data is long[].

I see. got it

kiszk · 2017-12-27T05:37:51Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+            case FloatType =>
+              val data = fromColumn.asInstanceOf[DoubleColumnVector].vector
+              var index = 0
+              while (index < batchSize) {


viirya · 2017-12-27T06:35:02Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

@@ -170,6 +171,8 @@ case class FileSourceScanExec(

  val needsUnsafeRowConversion: Boolean = if (relation.fileFormat.isInstanceOf[ParquetSource]) {
    SparkSession.getActiveSession.get.sessionState.conf.parquetVectorizedReaderEnabled
+  } else if (relation.fileFormat.isInstanceOf[OrcFileFormat]) {
+    SparkSession.getActiveSession.get.sessionState.conf.orcVectorizedReaderEnabled


Different than Parquet, for now we enable vectorized ORC reader when batch output is supported. We don't need unsafe row conversion at all for ORC. Because once it supports batch, we go batch-based approach. If it doesn't support batch, we don't enable vectorized ORC reader at all, so we don't need unsafe row conversion too.

Once we can enable vectorized ORC even batch is not supported, we need to add this.

Yep. I'll revert the change on this file, @viirya .

viirya · 2017-12-27T06:38:14Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+  /**
+   * Record reader from row batch.
+   */
+  private var rows: org.apache.orc.RecordReader = _


recordReader or rowReader?

Yep. It's renamed to recordReader.

viirya · 2017-12-27T06:39:12Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+  private var columnarBatch: ColumnarBatch = _
+
+  /**
+   * Writable columnVectors of ColumnarBatch.


columnVectors -> ColumnVector

Since it's Seq[WritableColumnVector], I'll keep the current one.

so it should be Writable column vectors

viirya · 2017-12-27T06:45:53Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+   * by copying from ORC VectorizedRowBatch columns to Spark ColumnarBatch columns.
+   */
+  private def nextBatch(): Boolean = {
+    if (rowsReturned >= totalRowCount) {


Should we add an assert to make sure batch is initialized (not null)?

Like Parquet's nextBatch, let's skip that assertion here.

viirya · 2017-12-27T06:51:40Z

...e/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.scala

+      val toColumn = columnVectors(i)
+
+      if (requestedColIds(i) < 0) {
+        toColumn.appendNulls(batchSize)


The column ids not requested are fixed. We don't need to reset them and re-append nulls into them again for each batch. We only need to append nulls into them once.

The column ids not requested are fixed is done by OrcDeserializer.

I meant, the columns not requested are not changed for each batch. For now every time you reset the batch and then re-append nulls into such columns. But those columns are not changed across different batches.

Once #20116 is merged, we reset by column vectors, not the columnar batch. So we can just reset the column vectors of the required columns.

I see, @viirya .

dongjoon-hyun · 2018-01-09T03:15:34Z

Oops. Without noticing your comments, I pushed another refactoring which split the functions.

henrify · 2018-01-09T03:42:00Z

@dongjoon-hyun Thank you for testing the split methods. If anything the benchmark results look couple of percent slower now? Oh well, at least it is good to know that your code is as fast as it can be!

I have no further ideas how performance could possibly be improved. Just many thanks to you and all reviewers for your hard work on this PR!

dongjoon-hyun · 2018-01-09T03:44:30Z

Thank you for your help, @henrify . I think it's within margin of deviation.
Split methods will be better for maintenance. So, I pushed it.

cloud-fan · 2018-01-09T04:17:31Z

...ore/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java

+  // The number of rows read and considered to be returned.
+  private long rowsReturned = 0L;
+
+  private long totalRowCount = 0L;


Do we still need these 2? They are not used for progress reporting anymore, and we don't rely on them to detect the end of file.

Right. We can remove them.

cloud-fan · 2018-01-09T04:30:07Z

...ore/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java

+    }
+  }
+
+  private void putNonNullBytes(int count, LongColumnVector fromColumn, WritableColumnVector toColumn) {


shall we revert these? Since they do not help with performance, I'd like to inline these small functions.

Yep. I'll revert them.

It almost seemed that they hurt the performance. The MR and Hive tests were up and down randomly as expected, but Vectorized tests were down in almost every benchmark.

cloud-fan · 2018-01-09T04:31:23Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcReadBenchmark.scala

+        ------------------------------------------------------------------------------------------------
+        Native ORC MR                                 3364 / 3391          0.3        3208.3       1.0X
+        Native ORC Vectorized                          273 /  284          3.8         260.1      12.3X
+        Hive built-in ORC                              831 /  842          1.3         792.8       4.0X


don't forget this question :) #19943 (comment)

Sorry for late response.

As we know, ORC read path evolves in two ways; ORC MR reader and ORC VectorizedRowBatch. But, the core part is ORC VectorizedRowBatch. ORC MR reader is reimplemented as a wrapper of the ORC VectorizedRowBatch.

What I mentioned previously was the difference between ORC VecterizedRowBatch and ORC MR reader (not about Hive 1.2.1 ORC read path). Maybe, the performance difference between Hive 1.2.1 ORC read path and ORC MR reader is due to the redevelopment of ORC MR reader.

Anyway, I consider both ORC MR and ORC Vectorized version as a single Native ORC. So, I described in a vague way sometimes. If this PR is merged, Spark is going to have both native ORC implementations (MR/Vectorized). Like Parquet, we will call vectorized version as Spark's native ORC.

What does Hive ORC use? The MR reader or vectorized reader or something else?

Now, the latest Hive shares the same library with Spark and the default is vectorized reader. That is the main goal of my efforts about Apache ORC 1.4.1.

This reverts commit 91b3d66.

This reverts commit 8fc2162.

SparkQA · 2018-01-09T04:52:58Z

Test build #85823 has finished for PR 19943 at commit 8fc2162.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-01-09T05:06:40Z

~~It's strange. The two tests have been passed in local labtop while jenkins always seems to fail. I'll investigate it.~~

Since 91b3d66, it seems to pass again.

[info] DataSourceWithHiveMetastoreCatalogSuite:
[info] - Persist non-partitioned parquet relation into metastore as managed table (3 seconds, 538 milliseconds)
[info] - Persist non-partitioned parquet relation into metastore as external table (1 second, 542 milliseconds)

SparkQA · 2018-01-09T06:39:06Z

Test build #85827 has finished for PR 19943 at commit 91b3d66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-09T07:14:58Z

Test build #85829 has finished for PR 19943 at commit db02555.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-09T08:05:02Z

Test build #85837 has finished for PR 19943 at commit 2cf98b6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-01-09T08:25:33Z

retest this please

viirya

LGTM with minor comments.

viirya · 2018-01-09T09:47:01Z

...ore/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java

+public class OrcColumnarBatchReader extends RecordReader<Void, ColumnarBatch> {
+
+  /**
+   * The default size of batch. We use this value for both ORC and Spark consistently


nit: We use this value for ORC reader to make it consistent with Spark's columnar batch, because their default batch sizes are different like the following.

viirya · 2018-01-09T09:53:51Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala

+      schema.length <= conf.wholeStageMaxNumFields &&
+      schema.forall(_.dataType.isInstanceOf[AtomicType])
+  }
+


Do we need to implement vectorTypes as ParquetFileFormat? Seems it affects the actual columnar vector types used in generated codes, instead of general ColumnVector. May it affect performance a bit?

SparkQA · 2018-01-09T11:37:48Z

Test build #85845 has finished for PR 19943 at commit 2cf98b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-01-09T13:49:05Z

thanks, merging to master/2.3! Let's address the comments in follow-up.

BTW @dongjoon-hyun let's keep our discussion on #19943 (comment)

## What changes were proposed in this pull request? This PR adds an ORC columnar-batch reader to native `OrcFileFormat`. Since both Spark `ColumnarBatch` and ORC `RowBatch` are used together, it is faster than the current Spark implementation. This replaces the prior PR, #17924. Also, this PR adds `OrcReadBenchmark` to show the performance improvement. ## How was this patch tested? Pass the existing test cases. Author: Dongjoon Hyun <[email protected]> Closes #19943 from dongjoon-hyun/SPARK-16060. (cherry picked from commit f44ba91) Signed-off-by: Wenchen Fan <[email protected]>

dongjoon-hyun · 2018-01-09T15:12:13Z

Thank you so much, @cloud-fan , @mmccline , @viirya , @henrify , @kiszk , @HyukjinKwon !
I'll proceed to follow-ups.

henrify · 2018-01-09T21:45:00Z

Great job guys! Also, check through the spam of your public github email address for a small gift @dongjoon-hyun @cloud-fan @viirya @kiszk @HyukjinKwon @mmccline

ghost reviewed Dec 11, 2017

View reviewed changes

viirya reviewed Dec 12, 2017

View reviewed changes

dongjoon-hyun mentioned this pull request Dec 21, 2017

[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive and sql/core #17980

Closed

cloud-fan reviewed Dec 22, 2017

View reviewed changes

kiszk reviewed Dec 27, 2017

View reviewed changes

viirya reviewed Dec 27, 2017

View reviewed changes

Split functions.

8fc2162

Address comments.

91b3d66

Use recordReader.getProgress

db02555

cloud-fan reviewed Jan 9, 2018

View reviewed changes

dongjoon-hyun added 2 commits January 8, 2018 20:34

Revert "Address comments."

8abbbc3

This reverts commit 91b3d66.

Revert "Split functions."

42182e3

This reverts commit 8fc2162.

Remove totalRowCount/rowsReturned and address comments..

2cf98b6

viirya approved these changes Jan 9, 2018

View reviewed changes

viirya reviewed Jan 9, 2018

View reviewed changes

asfgit closed this in f44ba91 Jan 9, 2018

dongjoon-hyun deleted the SPARK-16060 branch January 9, 2018 15:12

dongjoon-hyun mentioned this pull request Jan 11, 2018

[SPARK-21783][SQL] Turn on ORC filter push-down by default #18991

Closed

[SPARK-16060][SQL] Support Vectorized ORC Reader #19943

[SPARK-16060][SQL] Support Vectorized ORC Reader #19943

Conversation

dongjoon-hyun commented Dec 11, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Dec 12, 2017

Choose a reason for hiding this comment

dongjoon-hyun Dec 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Dec 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Dec 12, 2017

SparkQA commented Dec 12, 2017

SparkQA commented Dec 12, 2017

dongjoon-hyun commented Dec 12, 2017

SparkQA commented Dec 13, 2017

dongjoon-hyun commented Dec 13, 2017

cloud-fan commented Dec 13, 2017

dongjoon-hyun commented Dec 13, 2017

cloud-fan commented Dec 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henrify commented Dec 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Jan 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Dec 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya Dec 30, 2017 • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Dec 11, 2017 •

edited

Loading

dongjoon-hyun Dec 12, 2017 •

edited

Loading

dongjoon-hyun Dec 12, 2017 •

edited

Loading

dongjoon-hyun Jan 6, 2018 •

edited

Loading

dongjoon-hyun Dec 28, 2017 •

edited

Loading

viirya Dec 30, 2017 •

edited

Loading