[BUG] Dynamic partitions metric for insert into hive appears to be off

[revans2]

Describe the bug
We recently saw a customer query where the number of dynamic partitions was 30,000 but the CPU had it at 20. I was not able to dig into the details of it, but it is clearly wrong, or our code is doing something very bad.

[mattahrens]

@winningsix: can you give any more details about this scenario?

[winningsix]

@mattahrens We do see a case in one of customers logs. But it's still needed to understand its impact to the overall performance. So we may circle back this when performance impact confirmed.

cc @firestarman

[firestarman]

Better to have a repro case to understand what will happen here.
AFAIK, GPU has a different count algorithm for dynamic partitions from CPU. But I am not sure whether it is related.
CPU: Add the partition value to a Set, and get the Set size as the number of dynamic partitions.
GPU: Always increase the number by one after dectecting a different partition.

[jlowe]

The GPU computation is incorrect -- it needs to match the CPU behavior. This is an existing Spark metric, we need to match the semantics of the metric if we're going to report it.

Note that the CPU is combining all of the reported partitions into a set on the driver, not within each task. I suspect the GPU is simply reporting partition counts per task, which means we're probably over counting partitions when multiple tasks end up writing to the same partition.