`/notifications` returning stale outdated results

[MadLittleMods]

Description

As reported in element-hq/element-web#24616 (see "2. When the notification panels shows some old outdated notifications"), /notifications appears to be returning stale, outdated results.

When opening the notification panel in Element, it makes a request to https://gitter.ems.host/_matrix/client/r0/notifications?limit=20&only=highlight which returns some results (only 7) and these are from September 2021. And it also makes another paginated request, https://gitter.ems.host/_matrix/client/r0/notifications?limit=20&only=highlight&from=6554474 using the next_token from the previous request for the ?from in this request.

1: https://gitter.ems.host/_matrix/client/r0/notifications?limit=20&only=highlight

Response

{
	"notifications": [
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1631394783651,
			"event": {
				"content": {
					"body": "> <@elliria-5e77940ad73408ce4fdd9fb6:gitter.im> I messed around with it extensively today to get all the details for a bug report and found that it can't be consistently repeated, but only happens some of the time. Should I still file it if it's inconsistent?\n\nYup",
					"format": "org.matrix.custom.html",
					"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!dqBkUNHOHmuFWvsiHH:gitter.im/$CjLS68tmWZ2q1gO5iU69cw-Oxq3wleZjKKm0dPmi25o?via=kde.org&via=gitter.im&via=matrix.org\">In reply to</a> <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">@elliria-5e77940ad73408ce4fdd9fb6:gitter.im</a><br />I messed around with it extensively today to get all the details for a bug report and found that it can&#39;t be consistently repeated, but only happens some of the time. Should I still file it if it&#39;s inconsistent?</blockquote></mx-reply>Yup",
					"m.relates_to": {
						"m.in_reply_to": {
							"event_id": "$CjLS68tmWZ2q1gO5iU69cw-Oxq3wleZjKKm0dPmi25o"
						}
					},
					"msgtype": "m.text"
				},
				"origin_server_ts": 1631394783403,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 45707853657
				},
				"event_id": "$1YeLgxM_rdcx1cSpTFKBobbJbLVeapUUQvR3oyMHrb4"
			},
			"read": true
		},
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1631394775876,
			"event": {
				"content": {
					"body": "> <@elliria-5e77940ad73408ce4fdd9fb6:gitter.im> Yep, that's one of the two methods I tried. I tried PIP and Git. I'd love complete step-by-step instructions from someone who uses Kubuntu from start to finish, with finish being AutoKey opened on the desktop and ready to go, because so far it's making me feel like a complete idiot. I'll give it one more go to see if I can solve it on my own, but am going to take a break for the moment and get away from it.\n\nHow did pip go wrong?",
					"format": "org.matrix.custom.html",
					"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!dqBkUNHOHmuFWvsiHH:gitter.im/$6-Uox14OfobfZV6KD1If7imprr7KM9ipszTANN9kGCY?via=kde.org&via=gitter.im&via=matrix.org\">In reply to</a> <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">@elliria-5e77940ad73408ce4fdd9fb6:gitter.im</a><br />Yep, that&#39;s one of the two methods I tried. I tried PIP and Git. I&#39;d love complete step-by-step instructions from someone who uses Kubuntu from start to finish, with finish being AutoKey opened on the desktop and ready to go, because so far it&#39;s making me feel like a complete idiot. I&#39;ll give it one more go to see if I can solve it on my own, but am going to take a break for the moment and get away from it.</blockquote></mx-reply>How did pip go wrong?",
					"m.relates_to": {
						"m.in_reply_to": {
							"event_id": "$6-Uox14OfobfZV6KD1If7imprr7KM9ipszTANN9kGCY"
						}
					},
					"msgtype": "m.text"
				},
				"origin_server_ts": 1631394775400,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 45707861660
				},
				"event_id": "$k5BUicg6rN27kzzs3UfWAU33WcIxo71v-2wSh7tJlAM"
			},
			"read": true
		},
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1631305498705,
			"event": {
				"content": {
					"body": "> <@elliria-5e77940ad73408ce4fdd9fb6:gitter.im> I'm glad you fixed it. As to the beta, I wrestled with it in a variety of ways yesterday to try to install and use it in a VM with no success. I like to think that I'm not an idiot, but it's defeating me. Do you (or does anyone) have fool-proof step-by-step instructions for successfully installing, updating, and running the beta and/or any of the other development versions (if there are others that can be run)?\n\n`pip install --pre autokey` is probably the easiest way, but then there are a few system dependecies you'll need",
					"format": "org.matrix.custom.html",
					"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!dqBkUNHOHmuFWvsiHH:gitter.im/$CSnHh_MqeKt_YfT3rzmmQOLakgdoqS9ueoCWUHXf9U0?via=kde.org&via=gitter.im&via=matrix.org\">In reply to</a> <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">@elliria-5e77940ad73408ce4fdd9fb6:gitter.im</a><br />I&#39;m glad you fixed it. As to the beta, I wrestled with it in a variety of ways yesterday to try to install and use it in a VM with no success. I like to think that I&#39;m not an idiot, but it&#39;s defeating me. Do you (or does anyone) have fool-proof step-by-step instructions for successfully installing, updating, and running the beta and/or any of the other development versions (if there are others that can be run)?</blockquote></mx-reply>`pip install --pre autokey` is probably the easiest way, but then there are a few system dependecies you'll need",
					"m.relates_to": {
						"m.in_reply_to": {
							"event_id": "$CSnHh_MqeKt_YfT3rzmmQOLakgdoqS9ueoCWUHXf9U0"
						}
					},
					"msgtype": "m.text"
				},
				"origin_server_ts": 1631305498539,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 45797138521
				},
				"event_id": "$45NYvSz6sBbqayexby5irf71_pWVUD_kM53PtdDeOh8"
			},
			"read": true
		},
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1630095821345,
			"event": {
				"content": {
					"body": "> <@elliria-5e77940ad73408ce4fdd9fb6:gitter.im> Hey there, everybody. There's an AutoKey behavior that has changed from version 0.09.4 to the current 0.95.10 and I'm not sure if it's a bug, an intentional fix, something interesting, something neutral, or something else entirely, so I figured I'd describe it here to see what you guys think and to find out if it should be filed as a bug. \n> \n> AutoKey 0.90.4 accepted ```---``` (three dashes) as an Abbreviation for a phrase or script.\n> \n> AutoKey 0.95.10 does not accept ```---``` (three dashes) as an Abbreviation unless you trick it. If you try to set that as the Abbreviation, AutoKey ignores your input entirely and does nothing. You can trick it by setting ```x---``` as the Abbreviation and, after it accepts it, editing the Abbreviation to remove the x. The three-dash Abbreviation then works without complaint.\n> \n> For the record, the environment for AutoKey 0.90.4 was Ubuntu MATE 16.04 LTS and the environment for AutoKey 0.95.10 was Kubuntu 20.04 LTS.\n> \n\nSounds like a bug to me, thank you. If possible could you file it as an issue? Otherwise I will if I get around to it",
					"format": "org.matrix.custom.html",
					"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!dqBkUNHOHmuFWvsiHH:gitter.im/$iaWuRfG2-op9mfTSR9JN7Mw9QAgKHhRV-bU5T-VhMs8?via=kde.org&via=gitter.im&via=matrix.org\">In reply to</a> <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">@elliria-5e77940ad73408ce4fdd9fb6:gitter.im</a><br /><p>Hey there, everybody. There&#39;s an AutoKey behavior that has changed from version 0.09.4 to the current 0.95.10 and I&#39;m not sure if it&#39;s a bug, an intentional fix, something interesting, something neutral, or something else entirely, so I figured I&#39;d describe it here to see what you guys think and to find out if it should be filed as a bug. </p><p>AutoKey 0.90.4 accepted <code>---</code> (three dashes) as an Abbreviation for a phrase or script.</p><p>AutoKey 0.95.10 does not accept <code>---</code> (three dashes) as an Abbreviation unless you trick it. If you try to set that as the Abbreviation, AutoKey ignores your input entirely and does nothing. You can trick it by setting <code>x---</code> as the Abbreviation and, after it accepts it, editing the Abbreviation to remove the x. The three-dash Abbreviation then works without complaint.</p><p>For the record, the environment for AutoKey 0.90.4 was Ubuntu MATE 16.04 LTS and the environment for AutoKey 0.95.10 was Kubuntu 20.04 LTS.</p></blockquote></mx-reply>Sounds like a bug to me, thank you. If possible could you file it as an issue? Otherwise I will if I get around to it",
					"m.relates_to": {
						"m.in_reply_to": {
							"event_id": "$iaWuRfG2-op9mfTSR9JN7Mw9QAgKHhRV-bU5T-VhMs8"
						}
					},
					"msgtype": "m.text"
				},
				"origin_server_ts": 1630095820959,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 47006816101
				},
				"event_id": "$TWt_e4KX_Dh3wV2DOJ-6EEPQPkED60GqctgaklNQyM8"
			},
			"read": true
		},
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1626726000582,
			"event": {
				"content": {
					"body": "Awesome work Elliria (Elliria) !",
					"format": "org.matrix.custom.html",
					"formatted_body": "Awesome work <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">Elliria (Elliria)</a> !",
					"msgtype": "m.text"
				},
				"origin_server_ts": 1626726000231,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 50376636829
				},
				"event_id": "$6mzVma5MPa4HFHyb-WmbFzODikdgb_gHIvc30Q5_wmY"
			},
			"read": true
		},
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1624489549448,
			"event": {
				"content": {
					"body": "> <@elliria-5e77940ad73408ce4fdd9fb6:gitter.im> That's encouraging. At the moment, I have a few projects I have to get through before I can wear my official coder hat, but I'll continue playing around with code in general and poking around in the AutoKey sources just to see if anything jumps out at me.\n\nI recommend looking at tests as a starting point. They show you how code should work.\n(this is general advice more that for Autokey since our testing is a bit spotty)",
					"format": "org.matrix.custom.html",
					"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!dqBkUNHOHmuFWvsiHH:gitter.im/$jZs3fG_Blnp7QxdCJ_QxhSYpjEUUTzVXE3mc5lsbB5Q?via=kde.org&via=gitter.im&via=matrix.org\">In reply to</a> <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">@elliria-5e77940ad73408ce4fdd9fb6:gitter.im</a><br />That&#39;s encouraging. At the moment, I have a few projects I have to get through before I can wear my official coder hat, but I&#39;ll continue playing around with code in general and poking around in the AutoKey sources just to see if anything jumps out at me.</blockquote></mx-reply>I recommend looking at tests as a starting point. They show you how code should work.<br />(this is general advice more that for Autokey since our testing is a bit spotty)",
					"m.relates_to": {
						"m.in_reply_to": {
							"event_id": "$jZs3fG_Blnp7QxdCJ_QxhSYpjEUUTzVXE3mc5lsbB5Q"
						}
					},
					"msgtype": "m.text"
				},
				"origin_server_ts": 1624489517264,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 52613119796
				},
				"event_id": "$yleRn95D6kE7gltkxObRzg2XPVi3aP2Hl8NN3Me9Ymo"
			},
			"read": true
		},
		{
			"room_id": "!dqBkUNHOHmuFWvsiHH:gitter.im",
			"profile_tag": null,
			"actions": [
				"notify",
				{
					"set_tweak": "sound",
					"value": "default"
				},
				{
					"set_tweak": "highlight"
				}
			],
			"ts": 1624414848819,
			"event": {
				"content": {
					"body": "> <@elliria-5e77940ad73408ce4fdd9fb6:gitter.im> Yep, it looks like that's where it is and... oof... that code is out of my league. I was hoping to see something simple and obvious that was wrong, but there are way too many references in there to things that could be the cause that I got all tangled up trying to follow them. Gotta stick with more basic code for now. All I can say is good luck. We're all counting on you.\n\nI also thought it would be simple, just adding/fixing a callback. But no, I think the issue is that callback for left click doesn't exist separate to right, or if it does it is documented poorly",
					"format": "org.matrix.custom.html",
					"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!dqBkUNHOHmuFWvsiHH:gitter.im/$MD02yOZilNhT64gDFmop5AkQXzmdrHFZKDNp9xO1j_s?via=kde.org&via=gitter.im&via=matrix.org\">In reply to</a> <a href=\"https://matrix.to/#/@elliria-5e77940ad73408ce4fdd9fb6:gitter.im\">@elliria-5e77940ad73408ce4fdd9fb6:gitter.im</a><br />Yep, it looks like that&#39;s where it is and... oof... that code is out of my league. I was hoping to see something simple and obvious that was wrong, but there are way too many references in there to things that could be the cause that I got all tangled up trying to follow them. Gotta stick with more basic code for now. All I can say is good luck. We&#39;re all counting on you.</blockquote></mx-reply>I also thought it would be simple, just adding/fixing a callback. But no, I think the issue is that callback for left click doesn't exist separate to right, or if it does it is documented poorly",
					"m.relates_to": {
						"m.in_reply_to": {
							"event_id": "$MD02yOZilNhT64gDFmop5AkQXzmdrHFZKDNp9xO1j_s"
						}
					},
					"msgtype": "m.text"
				},
				"origin_server_ts": 1624414848530,
				"sender": "@silico_biomancer:kde.org",
				"type": "m.room.message",
				"unsigned": {
					"age": 52687788530
				},
				"event_id": "$kcf9okrQvjZyud6FHH8pDDN05_xlSxB49baxYYcv9U8"
			},
			"read": true
		}
	],
	"next_token": "6554474"
}

2: https://gitter.ems.host/_matrix/client/r0/notifications?limit=20&only=highlight&from=6554474

Response

{
	"notifications": [],
	"next_token": null
}

---

One aspect that could be playing a role is we had push.enable_push disabled for a little while during the Gitter import and then re-enabled it before everyone onboarded to the platform. Is this some stream position stuck problem for some users?

/notifications appears to work for me but we have reports from a couple of users that this isn't working (and as evidenced by their request/response above).

I've sent @elliria-5e77940ad73408ce4fdd9fb6:gitter.im direct mentions in a public, unencrypted room to doubly make sure this isn't some misunderstanding in how /notifications works (because it doesn't include encrypted events for example).

Affected users:

• @elliria-5e77940ad73408ce4fdd9fb6:gitter.im
• @josephj11-5a3f34bfd73408ce4f84a5df:gitter.im

Steps to reproduce

1. Call /notifications?limit=20&only=highlight to get the latest events that mention you
2. Notice stale results that don't include your latest mentions in a public unencrypted room.

Homeserver

gitter.im

Synapse Version

1.77.0

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgreSQL

Workers

Multiple workers

Platform

EMS

Configuration

No response

Relevant log output

Do we actually log anything useful? The request log doesn't seem that helpful.

If you want to dive in, see the logs in Kibana from EMS

Anything else that would be useful to know?

No response

[clokep]

Disabling notifications means they are literally not calculated (not that they just aren't pushed). This sounds like it is working as expected? Only notifications from the "live" timeline are appearing? Disabling notifications also means that no notifications in the "live" room will have been calculated during that time period.

[MadLittleMods]

Disabling notifications

Push notifications should be re-enabled on gitter.im. I'm not sure exactly where you are coming to that conclusion? Does the user have notifications disabled or something (did you check the database)?

Only notifications from the "live" timeline are appearing? Disabling notifications also means that no notifications in the "live" room will have been calculated during that time period.

This doesn't seem applicable to the situation. The person in question is missing notifications from me today in a public, unencrypted room.

Whereas with my account, I see mentions from testing today.

[clokep]

This doesn't seem applicable to the situation. The person in question is missing notifications from me today in a public, unencrypted room.

I must have misunderstood the original report then, if they're re-enabled it shouldn't applicable.

Note that endpoint only returns highlight notifications. I can debug more tomorrow and check the database.

[clokep]

I've sent @elliria-5e77940ad73408ce4fdd9fb6:gitter.im direct mentions in a public, unencrypted room to doubly make sure this isn't some misunderstanding in how /notifications works (because it doesn't include encrypted events for example).

Do you have example room IDs & event IDs that you expected to be highlight notifications? (Would also be interesting to confirm whether those did appear as red notifications in Element!)

Looking in the database for @elliria-5e77940ad73408ce4fdd9fb6:gitter.im there's only 7 rows all from 2021, so what's being returned matches what's in the database (always a good thing!)

[MadLittleMods]

@clokep The room is !lwgIRYKXLawruYybjr:matrix.org and the test mentions to @elliria-5e77940ad73408ce4fdd9fb6:gitter.im were in the following events:

• $qw1XFLKv15POhIRK054AYPb0QmaMH7DaL87u6va2IDU
• $2oxpfOnM9lU2LZ1JI__Ex_pRdQoy815LaoR-3D_z2Ds
• $LVBafWYSLES2ij160_AI-4Bga2S-Fqshx84-6S_2jV4

[clokep]

I can't be certain things haven't changed since your test, but the @elliria-5e77940ad73408ce4fdd9fb6:gitter.im user has the .m.rule.contains_display_name and .m.rule.contains_user_name set to:

[
  "notify",
  {
    "set_tweak": "highlight",
    "value": false
  }
]

With this configuration they will get notified, but not highlighted (i.e gray counts, not red counts).

My guess is that the first two options under Notifications > Mentions & keywords are set to "On" instead of "Noisy":

---

Looking at the notifications in the database I do see one event in that room which is a notification for this user (event ID $ETx3XPCEzHMzwIbZHuSCqrg5JCTDBk7CY2d02dvaF0E, the content does contain a mention). This is from 2023-02-23 19:44:38+Z so almost 24 hours ago -- we delete non-highlight notifications after 1 day IIRC (keeping only a summary of those notifications). The given event IDs are from > 24 hours ago (they're all from 2023-02-22).

---

I think this is just do to configuration, so I'm going to close this, but please shout if it seems there's something else going on.

[MadLittleMods]

Oh man, Element notification settings are crazy obtuse (or vice versa against Matrix push rules). I would expect On to still notify me just without sound bings and be considered a "highlight".

@clokep Thanks for looking into this and confirming the actual user settings from the database source of truth!

I hope some aspect of this is being addressed in the GetYourUpdates push to remove this confusion.

---

As an aside, maybe that's the key ingredient to get keyword mentions showing up in the notification panel, element-hq/element-meta#1212. I just need to set my Mentions containing keywords setting to Noisy

[clokep]

I would expect On to still notify me just without sound bings and be considered a "highlight".

Yeah, it is quite confusing. 😢

As an aside, maybe that's the key ingredient to get keyword mentions showing up in the notification panel, element-hq/element-meta#1212. I just need to set my Mentions containing keywords setting to Noisy

That sounds quite likely actually! 👍