synapse_forward_extremities metric appears to have no rooms in it for periods of time

[benbz]

So it appears like Synapse is reporting there are 0 rooms with <= Infinity Forward Extremities. This appears to happen semi-regularly

[clokep]

That's a bit odd, is this new behavior or has it been going on for a while?

Note that this value is only updated every 60 minutes (updated by the read_forward_extremities background task), so that's why it seems to oscillate between the two values.

[benbz]

The graphs I have access to only go back 30 days. I believe I can see this behaviour going on for the whole of that period, but it is difficult to tell

[anoadragon453]

Assuming this isn't due to performance or configuration changes, the only way I could see this being possible is if the query for this metric returning zero rows:

synapse/synapse/storage/databases/main/metrics.py

Lines 66 to 89 in 437a99f

	@wrap_as_background_process("read_forward_extremities")
	async def _read_forward_extremities(self):
	def fetch(txn):
	txn.execute(
	"""
	SELECT t1.c, t2.c
	FROM (
	SELECT room_id, COUNT(*) c FROM event_forward_extremities
	GROUP BY room_id
	) t1 LEFT JOIN (
	SELECT room_id, COUNT(*) c FROM current_state_events
	GROUP BY room_id
	) t2 ON t1.room_id = t2.room_id
	"""
	)
	return txn.fetchall()
	res = await self.db_pool.runInteraction("read_forward_extremities", fetch)
	_extremities_collecter.update_data(x[0] for x in res)
	_excess_state_events_collecter.update_data(
	(x[0] - 1) * x[1] for x in res if x[1]
	)

Handing [] to _extremities_collecter.update_data will clear all values.

@benbz is the synapse_excess_extremity_events bucket showing similar behaviour btw?

[benbz]

synapse_excess_extremity_events does seem to exhibit the same behaviour.

One thing I will say is that all the examples I saw in the last 48 hours seem to correspond to a server restart. And metrics only start being non-zero an hour later. I can't confirm/deny whether the historical ones all relate to a server restart

[anoadragon453]

Aha! Well that bucket won't be updated until at least 1 hour has passed since the server has started up, so on any scrape in the meantine we'd see 0's.

One solution is to just call the function immediately, and then again every 1hr. Looks like we actually do this already for some metrics:

synapse/synapse/app/phone_stats_home.py

Lines 175 to 177 in fe0f4a3

	if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
	generate_monthly_active_users()
	clock.looping_call(generate_monthly_active_users, 5 * 60 * 1000)

[richvdh]

One solution is to just call the function immediately, and then again every 1hr.

It might be a bit annoying to have to call that relatively expensive query during startup. can we instead just not emit the metric until the query has run?