backfill data fails w/ Signature check failed

[ewingrj]

Description

I recently setup a homeserver. When I join #matrix-dev:matrix.org or #homeowners:matrix.org, my server is not able to backfill the room history.

I see multiple log entries along the lines of Failed to backfill from matrix.org because 401: Invalid signature for server half-shot.uk with key.

1 event it is failing for is 2018-01-17 14:46:16,780 - synapse.federation.federation_base - 160 - WARNING - GET-1765- Signature check failed for $15158439492625roDBL:half-shot.uk.

Talking to @Half-Shot, he moved his server the other day and generated new keys. His verify_keys can be found at https://matrix.org/_matrix/key/v2/query/half-shot.uk. The key used to sign the event was his old_verify_key.

I first joined one of these rooms after the keys were changed.

Something to note is that the expired_ts for the key was past. Half-Shot changed the expired_ts to a future date to see if that fixed the bug. As of now, it hasn't but I'm wondering if there is a cache somewhere. I deleted half-shot.uk server keys from the db & restart, and got the same error.

Steps to reproduce

• For bugs, list the steps
• that reproduce the bug
• using hyphens as bullet points

Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from homeserver.log, please include
those here (please be careful to remove any personal or private data):

Version information

• Homeserver: giveth.io

If not matrix.org:

• Version: Synapse/0.26.0 What version of Synapse is running?
• Install method: debain package manager
• Platform: VPS running debian 9.

[ewingrj]

okay, this is still an issue for me even after the cache has expired, as the half-shot keys query above reflects the new expired_ts.

[ara4n]

This is a mess, and related to https://github.com/matrix-org/matrix-doc/issues/619 and https://github.com/matrix-org/matrix-doc/issues/831. Basically, we need a way to namespace different generations of servers so that new incarnations don't get queried for old incarnations' keys.

That said, it really shouldn't end up with you being unable to backfill, so keeping this open to track that particular failure mode.

[richvdh]

this is a combination of #4012 and #3121