Planning performance regression after replacing Vec<PhysicalExpr> with HashSet<PhysicalExpr>.

[Weijun-H]

Describe the bug

          Hmm, I seem to see a slight slow down in planning 🤔  maybe worth some profiling

++ critcmp main refactor_hashset_exprs
group                                         main                                   refactor_hashset_exprs
-----                                         ----                                   ----------------------
logical_aggregate_with_join                   1.00  1471.3±32.58µs        ? ?/sec    1.01  1484.4±20.25µs        ? ?/sec
logical_select_all_from_1000                  1.00      5.2±0.02ms        ? ?/sec    1.00      5.2±0.03ms        ? ?/sec
logical_select_one_from_700                   1.00  1177.1±16.70µs        ? ?/sec    1.00  1172.2±16.21µs        ? ?/sec
logical_trivial_join_high_numbered_columns    1.00  1137.3±16.03µs        ? ?/sec    1.01  1144.0±13.67µs        ? ?/sec
logical_trivial_join_low_numbered_columns     1.00  1122.4±16.65µs        ? ?/sec    1.02  1144.8±25.55µs        ? ?/sec
physical_intersection                         1.00      2.4±0.04ms        ? ?/sec    1.00      2.4±0.04ms        ? ?/sec
physical_join_consider_sort                   1.00      3.2±0.02ms        ? ?/sec    1.00      3.3±0.02ms        ? ?/sec
physical_join_distinct                        1.01  1122.7±16.06µs        ? ?/sec    1.00  1117.1±17.36µs        ? ?/sec
physical_many_self_joins                      1.00     17.0±0.09ms        ? ?/sec    1.00     17.0±0.11ms        ? ?/sec
physical_plan_clickbench_all                  1.00    225.4±1.82ms        ? ?/sec    1.01    227.7±1.99ms        ? ?/sec
physical_plan_clickbench_q1                   1.00      3.3±0.04ms        ? ?/sec    1.01      3.3±0.05ms        ? ?/sec
physical_plan_clickbench_q10                  1.00      4.3±0.05ms        ? ?/sec    1.01      4.4±0.06ms        ? ?/sec
physical_plan_clickbench_q11                  1.00      4.5±0.06ms        ? ?/sec    1.01      4.5±0.05ms        ? ?/sec
physical_plan_clickbench_q12                  1.00      4.6±0.08ms        ? ?/sec    1.01      4.6±0.06ms        ? ?/sec
physical_plan_clickbench_q13                  1.00      4.2±0.05ms        ? ?/sec    1.01      4.2±0.06ms        ? ?/sec
physical_plan_clickbench_q14                  1.00      4.4±0.07ms        ? ?/sec    1.01      4.5±0.06ms        ? ?/sec
physical_plan_clickbench_q15                  1.00      4.3±0.06ms        ? ?/sec    1.03      4.4±0.17ms        ? ?/sec
physical_plan_clickbench_q16                  1.00      3.7±0.04ms        ? ?/sec    1.01      3.8±0.04ms        ? ?/sec
physical_plan_clickbench_q17                  1.00      3.8±0.06ms        ? ?/sec    1.01      3.9±0.05ms        ? ?/sec
physical_plan_clickbench_q18                  1.00      3.6±0.04ms        ? ?/sec    1.01      3.6±0.05ms        ? ?/sec
physical_plan_clickbench_q19                  1.00      4.5±0.05ms        ? ?/sec    1.01      4.5±0.06ms        ? ?/sec
physical_plan_clickbench_q2                   1.00      3.6±0.05ms        ? ?/sec    1.01      3.6±0.04ms        ? ?/sec
physical_plan_clickbench_q20                  1.00      3.3±0.05ms        ? ?/sec    1.01      3.4±0.04ms        ? ?/sec
physical_plan_clickbench_q21                  1.00      3.6±0.05ms        ? ?/sec    1.01      3.6±0.05ms        ? ?/sec
physical_plan_clickbench_q22                  1.00      4.6±0.08ms        ? ?/sec    1.01      4.6±0.07ms        ? ?/sec
physical_plan_clickbench_q23                  1.00      5.0±0.06ms        ? ?/sec    1.01      5.0±0.06ms        ? ?/sec
physical_plan_clickbench_q24                  1.00      5.8±0.07ms        ? ?/sec    1.01      5.8±0.09ms        ? ?/sec
physical_plan_clickbench_q25                  1.00      3.9±0.05ms        ? ?/sec    1.01      4.0±0.08ms        ? ?/sec
physical_plan_clickbench_q26                  1.00      3.6±0.06ms        ? ?/sec    1.02      3.7±0.05ms        ? ?/sec
physical_plan_clickbench_q27                  1.00      4.0±0.06ms        ? ?/sec    1.01      4.0±0.05ms        ? ?/sec
physical_plan_clickbench_q28                  1.00      4.7±0.07ms        ? ?/sec    1.02      4.8±0.11ms        ? ?/sec
physical_plan_clickbench_q29                  1.00      5.8±0.07ms        ? ?/sec    1.01      5.9±0.10ms        ? ?/sec
physical_plan_clickbench_q3                   1.00      3.5±0.05ms        ? ?/sec    1.02      3.6±0.04ms        ? ?/sec
physical_plan_clickbench_q30                  1.00     16.6±0.21ms        ? ?/sec    1.00     16.6±0.18ms        ? ?/sec
...

Originally posted by @alamb in #13612 (comment)

To Reproduce

No response

Expected behavior

No response

Additional context

No response