Support Spark Column Stats #10659

huaxingao · 2024-07-08T18:30:23Z

Co-authored-by: Huaxin Gao
Co-authored-by: Karuppayya Rajendran

This PR adds the column stats support, so Iceberg can report column stats to Spark engine for CBO.

Co-authored-by: Karuppayya Rajendran <[email protected]>

huaxingao · 2024-07-08T19:46:06Z

cc @szehon-ho @karuppayya

jeesou · 2024-07-10T15:46:46Z

Hi @huaxingao , @karuppayya, just to clear out a few aspects,
This PR would be a continuation of #10288.

To the above mentioned PR we would need to create a Procedure to trigger the Analyze action.

But would this PR changes mean that we would be able to see other columnar statistics like min, max, etc along with NDV in the .stat file?

Could you please help understand how "Iceberg can report column stats to Spark engine for CBO".

Because even with this PR changes along with #10288, the Statistics section looks like :
"blob-metadata" : [ {
"type" : "apache-datasketches-theta-v1",
"snapshot-id" : 6563878496533098178,
"sequence-number" : 1,
"fields" : [ 3 ],
"properties" : {
"ndv" : "3"
}
}]

Other statistics are not coming right now, which is expected as per the code.
Please help understand that how will we generate the other statistics and how will Spark access the statistics.

karuppayya · 2024-07-10T19:41:39Z

@jeesou I will be creating a PR for the procedure for Analyze action, when #10288 is merged .
Currently the spec supports only NDV. For more stats we will need to make spec chnages and other corresponding chnages.

This changes helps reporting stats to Spark using the DSv2 APIs, which is subsequently used in Join estimation, Dynamic Partition Pruning etc

singhpk234

@huaxingao @karuppayya want to know you thoughts on :
1/ sparks stats estimator for non CBO is super bad for v2 sources some additional info which sources like iceberg can provide for ex: rowCount gets simply ignored by them as. Do you have any thing in your mind in context of that I attempted this via :

[SPARK-39678][SQL] Improve stats estimation for v2 tables spark#37083
It didn't get any traction, what are your thoughts in this context

2/ Do you have perf numbers on how does this complements CBO performance considering CBO is disabled by default in OSS.

3/ Do we want to give users an options to extract stuff from sketches to extract extra stats rather than relying on precomputed ndv just ?

singhpk234 · 2024-07-11T20:54:58Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+
+    Map<NamedReference, ColumnStatistics> map = Maps.newHashMap();
+
+    if (readConf.enableColumnStats()) {


[Q] NDV can't be used for non CBO cases, do we need to couple this flag by checking cbo is enabled addtionally as well spark.sql.cbo.enabled other wise SizeInBytesOnlyStatsPlanVisitor will be used ?

Added the check for CBO. Thanks!

szehon-ho

Left a first round of comments

szehon-ho · 2024-07-11T18:58:00Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ColStats.java

+import org.apache.spark.sql.connector.read.colstats.ColumnStatistics;
+import org.apache.spark.sql.connector.read.colstats.Histogram;
+
+class ColStats implements ColumnStatistics {


Nit: this is a bit confusing, ColStats is just abbreviate the Spark class name. How about SparkColumnStatistics? (Looks like in the package there are a lot of class called SparkXXX in this situation)

Changed to SparkColumnStatistics. Thanks

szehon-ho · 2024-07-11T21:01:54Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+      return new Stats(0L, 0L, Maps.newHashMap());
+    }
+
+    Map<NamedReference, ColumnStatistics> map = Maps.newHashMap();


Can we have a more descriptive name?

Also can we avoid the allocation if enableColumnStats and CBO aren't enabled?

Changed to colStatsMap and also only allocate if enableColumnStats and CBO are enabled.

szehon-ho · 2024-07-11T21:02:47Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+
+        for (BlobMetadata blobMetadata : metadataList) {
+          long ndv = Long.parseLong(blobMetadata.properties().get("ndv"));
+          ColumnStatistics colStats = new ColStats(ndv, null, null, 0L, 0L, 0L, null);


Should we fill the other values from the manifest file?

In aggregate push down, I have some code to get min/max/nulls from the manifest file. I will extract the common code and reuse them. Will have a separate PR for this after this PR is merged. I have put a // TODO: Fill min, max and null from the manifest file

szehon-ho · 2024-07-11T21:05:01Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+        List<BlobMetadata> metadataList = (files.get(0)).blobMetadata();
+
+        for (BlobMetadata blobMetadata : metadataList) {
+          long ndv = Long.parseLong(blobMetadata.properties().get("ndv"));


I think we should be more defensive if there is no value, or numberformatexception?

Also check org.apache.iceberg.BlobMetadata#type?

I have added the code to check null and empty value. It's probably too strict to throw Exception if the value is null or empty String. I log a DEBUG message instead.

Is there an agreed type for org.apache.iceberg.BlobMetadata#type? If yes, I will add a check.

org.apache.iceberg.puffin.StandardBlobTypes has the supported types, but currently its just one

szehon-ho · 2024-07-11T21:05:59Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

@@ -97,6 +117,36 @@ public static Object[][] parameters() {
    };
  }

+  @BeforeAll


Why is it needed?

I don't think this is required, for throwing in new properties you should probably use

protected void withSQLConf(Map<String, String> conf, Action action) {

Removed and added withSQLConf

szehon-ho · 2024-07-11T21:07:51Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

@@ -90,4 +90,8 @@ private SparkSQLProperties() {}
  public static final String EXECUTOR_CACHE_LOCALITY_ENABLED =
      "spark.sql.iceberg.executor-cache.locality.enabled";
  public static final boolean EXECUTOR_CACHE_LOCALITY_ENABLED_DEFAULT = false;
+
+  // Controls whether column statistics are enabled when estimating statistics


the comment seems not meaningful at first read. Can we be more precise?

Changed to // Controls whether to calculate column statistics and report them to Spark

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

karuppayya · 2024-07-12T01:10:39Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+        List<BlobMetadata> metadataList = (files.get(0)).blobMetadata();
+
+        for (BlobMetadata blobMetadata : metadataList) {
+          long ndv = Long.parseLong(blobMetadata.properties().get("ndv"));


Also check org.apache.iceberg.BlobMetadata#type?

jeesou · 2024-07-15T17:20:59Z

@jeesou I will be creating a PR for the procedure for Analyze action, when #10288 is merged . Currently the spec supports only NDV. For more stats we will need to make spec chnages and other corresponding chnages.

This changes helps reporting stats to Spark using the DSv2 APIs, which is subsequently used in Join estimation, Dynamic Partition Pruning etc

Hi @karuppayya , could you please let us know where can we find this spec which will be changed to accommodate the other statistics.

And could you Share some knowledge on how to hit the DSv2 apis, while using pyspark, because the documentations don't have much clarity.

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ColStats.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkChangelogScan.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

karuppayya · 2024-07-17T19:03:53Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+        List<BlobMetadata> metadataList = (files.get(0)).blobMetadata();
+
+        for (BlobMetadata blobMetadata : metadataList) {
+          int id = blobMetadata.fields().get(0);


Should we check if there is more than one field here(for example, ndv stats is collected for say field1 and field2) and not propagate the stats if so?

I can't find detailed documentation for BlobMetadata in the spec. My understanding is that for the blob type apache-datasketches-theta-v1, ndv is required. I assume that ndv is set in the properties as shown in the following example. If my assumption is correct, it seems to me that we can only have one field in the fields, establishing a one-to-one relationship with the ndv in the properties.

new GenericBlobMetadata( APACHE_DATASKETCHES_THETA_V1, snapshotId, 1, ImmutableList.of(1), ImmutableMap.of("ndv", "4"))

cc @findepi

Correct, the apache-datasketches-theta-v1 should be calculated on one field.
And yes, there should be the ndv property set. The property may seem somewhat redundant within the Puffin file, but allow faster access to the information at SELECT-time. More importantly, the properties are propagated to the table metadata and so a query planner accesses the NDV information without opening the Puffin file at all.

karuppayya · 2024-07-17T19:08:15Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+          }
+
+          // TODO: Fill min, max and null from the manifest file
+          ColumnStatistics colStats = new SparkColumnStatistics(ndv, null, null, 0L, 0L, 0L, null);


When ndv is not set, we are sending in 0. Is that intended?

Yeah I didn't see notes in the Spark docs about what 0 would mean in this context.

Checked Spark code, should be None if ndv is not available. I changed the code accordingly.

Looks like it is still 0?

Changed to null.

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

huaxingao · 2024-07-19T01:37:19Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

@@ -189,9 +192,8 @@ protected Statistics estimateStatistics(Snapshot snapshot) {

    boolean cboEnabled =
        Boolean.parseBoolean(spark.conf().get(SQLConf.CBO_ENABLED().key(), "false"));
-    Map<NamedReference, ColumnStatistics> colStatsMap = null;
+    Map<NamedReference, ColumnStatistics> colStatsMap = Maps.newHashMap();


In Spark side, it has a columnStats().isEmpty, so I changed back to Maps.newHashMap() to avoid the NPE.

What I meant here was to have it be Collection EmptyMap unless cbo is enabled, probably a premature optimization. So feel free to ignore

Ie:

Map colStatsMap; if (readConf && cbo) colStatsMap = Maps.newHashMap() else colStatsMap = Collections.emptyMap()

Got it. Changed!

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkColumnStatistics.java

jeesou · 2024-07-26T07:51:50Z

Hi @huaxingao , @karuppayya
cc : @RussellSpitzer
We were running some tests on Spark with the latest codes.
We took the changes of the previous PR #10288, along with this PR changes, and generated the .stats file and we can see the ndv values in the metadata.

So to verify the performance enhancement we ran the TPCH queries. On running the query at 1000 G scale we are facing some issues on certain queries (query umbers - 5,7,8,9,10) where while performing the broadcast join, some error occurred.

I am sharing the log for query number 8 you can check.
I am sharing logical plan as well.

Sharing the error log for query 5 as well

Sharing the config we used for reference -

"spark.executor.cores": "6",
"spark.executor.memory": "24G",
"spark.driver.cores": "6",
"spark.driver.memory": "24G",
"spark.driver.maxResultSize":"0",
"spark.sql.iceberg.enable-column-stats": "true",
"spark.sql.cbo.enabled": "true",
"spark.sql.cbo.joinReorder.enabled": "true",

we have tried by upscaling the executor and driver cores and memoryto 12/48 scale. but received the same issue.

Kindly help us understand if we are missing anything out, or is this an issue.

saitharun15 · 2024-07-26T08:09:37Z

Hi @huaxingao , @karuppayya cc : @RussellSpitzer We were running some tests on Spark with the latest codes. We took the changes of the previous PR #10288, along with this PR changes, and generated the .stats file and we can see the ndv values in the metadata.

So to verify the performance enhancement we ran the TPCH queries. On running the query at 1000 G scale we are facing some issues on certain queries (query umbers - 5,7,8,9,10) where while performing the broadcast join, some error occurred.

I am sharing the log for query number 8 you can check. I am sharing logical plan as well.

Sharing the error log for query 5 as well

Sharing the config we used for reference -

"spark.executor.cores": "6", "spark.executor.memory": "24G", "spark.driver.cores": "6", "spark.driver.memory": "24G", "spark.driver.maxResultSize":"0", "spark.sql.iceberg.enable-column-stats": "true", "spark.sql.cbo.enabled": "true", "spark.sql.cbo.joinReorder.enabled": "true",

we have tried by upscaling the executor and driver cores and memoryto 12/48 scale. but received the same issue.

Kindly help us understand if we are missing anything out, or is this an issue.

We saw performance enhancement in couple of queries (1,2,19) where spark was able to perform broadcast join on larger tables where as previously without ndv stats, the plans were not that better. And also considering the 5th query when ran with hive with statistics and cbo enabled, it was not failing because spark used sort merge join instead of broadcast for larger tables, but incase of iceberg with ndv stats spark is using only broadcast join and failing due to errors:

" org.apache.spark.SparkException: Cannot broadcast the table that is larger than 8.0 GiB: 9.0 GiB. org.apache.spark.sql.errors.QueryExecutionErrors$.cannotBroadcastTableOverMaxTableBytesError and

: org.apache.spark.SparkUnsupportedOperationException: Can not build a HashedRelation that is larger than 8G.
org.apache.spark.sql.errors.QueryExecutionErrors$.cannotBuildHashedRelationLargerThan8GError"

RussellSpitzer · 2024-07-26T16:14:57Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

+        new SparkScanBuilder(spark, table, CaseInsensitiveStringMap.empty());
+    SparkScan scan = (SparkScan) scanBuilder.build();
+
+    Map<String, String> sqlConf1 =


sqlConf1 => reportStatsDisabled
sqlConf2 => reportStatsEnabled

Changed. Thanks

RussellSpitzer · 2024-07-26T16:24:32Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

@@ -734,6 +801,21 @@ private Expression[] expressions(Expression... expressions) {
    return expressions;
  }

+  private void checkStatistics(SparkScan scan, long expectedRowCount, boolean expectedColumnStats) {


This may be over parameterized, we only use this function 2 times and we use them for relatively different things. So I'm not sure it makes things clearer. It's also missing a bit to be completely parameterized.

So if it was completely parameterized we would want probably two functions like

checkStatisticsNotReported(scan) checkReportedStatistics(scan, rowCount, map<String, Int> distinctValueCounts)

As it is we hard code in the column name "id" and the distinct value count "4l" but parameterize the expectedRowCount. So we couldn't really re-use this function for any reason.

Now since we only actually use this function for 1 example above it may be ok to not even parameterize it at all. So I think there are 2 ways to go here.

Find more uses for the function and fully parameterize
Deparameterize and inline.

I think we have a few more use cases we should probably test out just for fun so maybe route 1 is better?

For example

Test where 1 column has NDV and the other does not.
Test where stats reporting is enabled, but the table does not have stats
Test with different distinct values for columns in the table

Thanks for the suggestion! Changed to route 1

huaxingao · 2024-07-26T21:34:52Z

@jeesou @saitharun15
Thanks for testing this out! The column stats are not yet accurate because I still need to retrieve the numOfNulls, min, and max from the manifest files. I will submit a follow-up PR to address this. Please wait for the follow-up PR and retest to see if the problem is resolved.

RussellSpitzer · 2024-07-26T21:36:29Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+            LOG.debug("DataSketch blob is not available for column {}", colName);
+          }
+
+          // TODO: Fill min, max and null from the manifest file


Can we remove the TODO and make an Issue?

Created issue

RussellSpitzer · 2024-07-26T21:39:15Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

+    expectedOneNDV.put("id", 4L);
+    withSQLConf(reportColStatsEnabled, () -> checkColStatisticsReported(scan, 4L, expectedOneNDV));
+
+    statisticsFile =


I'd split this one out into another test, minor nit

RussellSpitzer · 2024-07-26T21:39:34Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java

+    Map<String, String> reportColStatsEnabled =
+        ImmutableMap.of(SQLConf.CBO_ENABLED().key(), "true");
+
+    // Test table does not have col stats


I would put this one in another test too

RussellSpitzer

I have a few remaining test nits, but I think we are good here

sfc-gh-rspitzer · 2024-07-30T14:37:33Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+              .type()
+              .equals(org.apache.iceberg.puffin.StandardBlobTypes.APACHE_DATASKETCHES_THETA_V1)) {
+            String ndvStr = blobMetadata.properties().get(NDV_KEY);
+            if (ndvStr != null && !ndvStr.isEmpty()) {


We usually use the commons "!Strings.isNullOrEmpty" for these

Done. Thanks!

sfc-gh-rspitzer · 2024-07-30T14:39:22Z

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

+
+    boolean cboEnabled =
+        Boolean.parseBoolean(spark.conf().get(SQLConf.CBO_ENABLED().key(), "false"));
+    Map<NamedReference, ColumnStatistics> colStatsMap = null;


Thinking back to this change, may be cleaner to just set this to empty map and the reassign to Maps.newHashMap on 195 as is.

colStatsMap = Collections.emptyMap
if (report & cbo) {
colStatsMap = Maps.newHashMap
}

Fixed. Thanks!

RussellSpitzer · 2024-07-31T15:41:40Z

Thanks @huaxingao and @karuppayya ! This is a great addition to the Spark capabilities

huaxingao · 2024-07-31T16:00:51Z

Thanks a lot @RussellSpitzer! Also thanks to @szehon-ho @karuppayya @findepi @singhpk234 for helping with this PR!

Support Spark Column Stats

e37e5e2

Co-authored-by: Karuppayya Rajendran <[email protected]>

github-actions bot added the spark label Jul 8, 2024

karuppayya mentioned this pull request Jul 8, 2024

Spark Action to Analyze table #10288

Merged

singhpk234 reviewed Jul 11, 2024

View reviewed changes

szehon-ho reviewed Jul 11, 2024

View reviewed changes

karuppayya reviewed Jul 12, 2024

View reviewed changes

karuppayya mentioned this pull request Jul 12, 2024

Change the description in Table metadata spec about the cardinality/mapping between snapshot and puffin #10693

Open

6 tasks

RussellSpitzer reviewed Jul 15, 2024

View reviewed changes

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ColStats.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Jul 15, 2024

View reviewed changes

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkChangelogScan.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Jul 15, 2024

View reviewed changes

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Jul 15, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkScan.java Outdated Show resolved Hide resolved

huaxingao added 2 commits July 16, 2024 16:19

address comments

6d94588

spotless

1d61ca7

karuppayya reviewed Jul 17, 2024

View reviewed changes

RussellSpitzer reviewed Jul 17, 2024

View reviewed changes

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java Outdated Show resolved Hide resolved

address comments

1dd57bf

huaxingao commented Jul 19, 2024

View reviewed changes

remove static import

fe5cd0c

RussellSpitzer reviewed Jul 19, 2024

View reviewed changes

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Jul 19, 2024

View reviewed changes

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkColumnStatistics.java Show resolved Hide resolved

address comments

68b0ffb

RussellSpitzer reviewed Jul 26, 2024

View reviewed changes

address comments

7ff299f

RussellSpitzer reviewed Jul 26, 2024

View reviewed changes

RussellSpitzer approved these changes Jul 26, 2024

View reviewed changes

karuppayya approved these changes Jul 26, 2024

View reviewed changes

huaxingao mentioned this pull request Jul 26, 2024

Use Min, Max, and NumOfNulls from Manifest Files for Spark Column Stats #10791

Open

split tests

b677d32

sfc-gh-rspitzer reviewed Jul 30, 2024

View reviewed changes

address comments

6d154c0

RussellSpitzer merged commit 506fee4 into apache:main Jul 31, 2024
42 checks passed

huaxingao deleted the colstats branch July 31, 2024 16:00

jeesou mentioned this pull request Aug 28, 2024

Stale column stats getting reported when reading puffin files generated by Presto with Spark engine #11034

Closed

3 tasks

saitharun15 mentioned this pull request Nov 13, 2024

Spark 3.4: Support Spark Column Stats #11532

Merged

saitharun15 mentioned this pull request Nov 21, 2024

Spark : Derive Stats From Manifest on the Fly #11615

Open


		Map<NamedReference, ColumnStatistics> map = Maps.newHashMap();

		if (readConf.enableColumnStats()) {

Support Spark Column Stats #10659

Support Spark Column Stats #10659

Conversation

huaxingao commented Jul 8, 2024

huaxingao commented Jul 8, 2024

jeesou commented Jul 10, 2024

karuppayya commented Jul 10, 2024 • edited Loading

singhpk234 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szehon-ho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeesou commented Jul 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeesou commented Jul 26, 2024

saitharun15 commented Jul 26, 2024 • edited Loading

RussellSpitzer Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huaxingao commented Jul 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer commented Jul 31, 2024

huaxingao commented Jul 31, 2024

karuppayya commented Jul 10, 2024 •

edited

Loading

singhpk234 left a comment •

edited

Loading

saitharun15 commented Jul 26, 2024 •

edited

Loading

RussellSpitzer Jul 26, 2024 •

edited

Loading

RussellSpitzer Jul 26, 2024 •

edited

Loading