synapse loaded by lots of /federation/v1/event/... requests

[richvdh]

Every so often, my synapse goes off and spends a couple of hours spinning at 100% CPU. Investigation reveals that it is mostly spending that time processing /federation/v1/event/ requests - basically some other server in the federation starts hammering it for a load of events.

There are two questions here:

• why are other servers hammering it?
• it seems surprisingly expensive to serve the request. The CPU times are quite variable but some of them are over 200ms per request - which takes a while when there are tens of thousands of requests.

[richvdh]

anecdotally, the worst offenders seem to be on older synapses; in particular stephenson.cc which is on 0.18.7.