perf(dep_graph): Avoid allocating a set on when the number reads are … #69778

…small `reserve_and_rehash` takes up 1.4% of the runtime on the `packed-simd` benchmark which I believe is due to the number of reads are very low in many cases (see rust-lang#50565 for instance). This avoids allocating the set until we start allocating the `reads` `SmallVec` but it is possible that a lower limit might be better (not tested since the improvement will be hard to spot either way).