Rustdoc takes 8.5 GB of memory to document stm32h7

[jyn514]

Similar to but not the same as #78761: rustdoc uses an inordinate amount of memory to document large crates.

heaptrack_gui shows almost a third of the memory being allocated in get_blanket_impls:

However, the peak usage is actually in check_mod_item_types, which rustdoc can't do much about:

Rustdoc should work improving on both the total memory usage and peak memory usage.

This is meant as a tracking issue, since there are issues other than get_blanket_impls causing high memory use, such as #76382. There is a Zulip stream for technical discussion.

cc @adamgreig

[jyn514]

This is causing docs.rs to have to periodically bump the sandbox limits when documenting stm32 crates: rust-lang/docs.rs#1179.

[camelid]

P-medium: it's annoying that it takes so much memory, but it's not like you can't document it. And it's being worked on.

[jyn514]

I ran Rustdoc under DHAT and found that a good tenth of the allocations come from get_blanket_impls alone:

rust/src/librustdoc/clean/blanket_impl.rs

Line 130 in 2225ee1

items: self

. I'm not sure how to upload the full 81 MB file to GitHub, but feel free to ping me here or on Zulip, I'll keep a copy around Here's a link to Google Drive with the full file. Unfortunately DHAT takes a full 50 minutes to run rustdoc on stm32 so it's a pain to collect on your own.

[jyn514]

Another 7% just storing the structs themselves:

rust/src/librustdoc/passes/collect_trait_impls.rs

Line 128 in 2225ee1

self.impls.extend(get_auto_trait_and_blanket_impls(

. Maybe I should be asking just how many impls there are?

[jyn514]

Maybe I should be asking just how many impls there are?

+-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect-trait-impls                             | 33.31s    | 45.735          | 39.64s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| render_html                                     | 23.69s    | 32.537          | 23.69s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| <unknown>                                       | 6.81s     | 9.345           | 10.21s   | 2417746    |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| clean_crate                                     | 2.85s     | 3.913           | 4.01s    | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect-intra-doc-links                         | 918.47ms  | 1.261           | 970.03ms | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| expand_crate                                    | 905.65ms  | 1.244           | 909.95ms | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| build_extern_trait_impl                         | 796.51ms  | 1.094           | 1.38s    | 20080      |

The strange thing that most of the time is not spent on build_extern_trait_impl, it's somewhere else in collect_trait_impls.

[jyn514]

Almost all the time in collect_trait_impls is being spent on synthetic impls, specifically

rust/src/librustdoc/passes/collect_trait_impls.rs

Line 19 in 4031f7b

let mut krate = synth.fold_crate(krate);

+-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect_synthetic_impls                         | 32.71s    | 44.967          | 37.56s   | 1          |

[jyn514]

get_blanket_impls takes about 10x as long as get_synthetic_impls, but they're both pretty slow.

+-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| get_blanket_impls                               | 29.86s    | 41.053          | 34.48s   | 32176      |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| render_html                                     | 23.18s    | 31.879          | 23.18s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| <unknown>                                       | 6.92s     | 9.514           | 10.33s   | 2417746    |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| get_auto_trait_impls                            | 3.42s     | 4.704           | 3.69s    | 32176      |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| clean_crate                                     | 2.82s     | 3.871           | 3.98s    | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+

[jyn514]

This also impacts windows-rs.

[jyn514]

I profiled this recently, and it's down to 1.6 GB - still high, but a lot better than before. I'm going to close this until I find something more specific to track.