Faster joins: persist to database

[richvdh]

Database schema updates, and changes to room-join/event-persistence to record which rooms and events have partial state.

When we get a partial_state response from send_join, store information in the database about it:

• store a record about the room as a whole having partial state, and stash the list of member servers too.
• flag the join event itself as having partial state
• also, for any new events whose prev-events are partial-stated, note that they will also be partial-stated.

We don't yet make any attempt to interpret this data, so API calls (and a bunch of other things) are just going to get incorrect data.

Nor do we yet make any attempt to de-partial-state the events.

Part of my work on #11249.

[squahtx]

I'm unclear on why we need to do this again when de-outliering. Does de-outliering not imply that the event has already been through this code once?

[richvdh]

seems to be another great question. I'll try and remember what's going on here.

[richvdh]

Right, I've remembered what this is about.

Outliers have no state at all, so it's meaningless to say that they have "partial state". So, the first time they go through this code, we would not expect to set the partial-state flag.

When we de-outlier the event, it's possible that we might de-outlier it with only partial state, and in that case we need to add a row to partial_state_events when we de-outlier it, which is why I've put this code here.

I've added a robustness check to make sure it's true that outliers are never flagged with partial-state in 446fae6.

I've also got it on my todo list to make sure we end up with tests which include de-outliering events with partial state.

[squahtx]

Thanks for the explanation! So the outliers we persist will never be flagged with partial state because we always use EventContext.for_outlier() as the context*?

* with the exception of the outlier in RoomMemberMasterHandler._generate_local_out_of_band_leave for reasons that are beyond me right now.

[richvdh]

• with the exception of the outlier in RoomMemberMasterHandler._generate_local_out_of_band_leave for reasons that are beyond me right now.

Yeah, that looks a bit bogus. It means we end up recording state at the leave event, but also flagging it as an outlier, which seems wrong. Either it shouldn't be an outlier, or we shouldn't record the state at it. I'm going to park this for now though.

[richvdh]

for the record: I've ended up with a fix to this, at #12154, as it broke in the face of my fixes to #12074.

[squahtx]

lgtm, assuming my last question doesn't raise anything further