Multiregion movr workload performance discrepancies after primary key change

[rohany]

The demos here https://github.com/cockroachlabs/scripted-demos/tree/movr-20.1 set up the movr workload in 2 ways:

• Create the tables from the get go in a multi region fashion with partitions and zone configs
• Start the application in single region mode, and transition the schema to multi region using primary key changes.

There have been reported performance differences between the two -- p90, p99 latencies are 10-20 ms higher in the post primary key change state. We need to investigate what is causing the regression.

[rohany]

cc @awoods187

[awoods187]

@rohany I know you mentioned SFU as a possible cause here...not sure why that would be different on one cluster vs. another. cc @nvanbenschoten

What are our next steps?

[nvanbenschoten]

I'd recommend waiting until the fix for #49658 lands later today and then testing again.

[awoods187]

Looks like everything is merged--can we try it out now?

[nvanbenschoten]

Let's wait until #50173 is merged and backported (if you're running straight off release-20.1).

[nvanbenschoten]

That patch is now merged on master. The backport is here: #50216.

[rohany]

I reran these demos with these changes and could not replicate the performance differences with the related kv changes.